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AN INTRODUCTION TO EBCE 

ExperienM-Based Career Education (EBCE) is a f imdamantally diffarant 
type of education for secOTdary students* While students in 
tra^tional programs attend full-day classes at the high school, 
EBCE students spend a major portion of their time on learaing 
projects in the comiLaiity. Activitios in EBCE are tailored to 
individual needs , abilities, learning styles md goals, and students 
are guided in their learning tiirough ongoing relationships with a 
variety of working adults in the coMnmity. 

trough real world interactions with adults, EBCB students learn 
about careers, about life, ^out other people, about ttemselves. 
In addition, students learn the basic skills^^f critical thinking, 
science, personal and social davelopment , factional citiEenship 
and creative development. O^ay gain competence in the skills adults 
need to faction effectively in a teclmological society. ThBy learn 
to be responsible by helping desi^ their omi learning activities 
Bnd by following a set of account^ility standards that parallel 
th^ standards working adults are expected to maintain on the job. 

Perhaps most importantly, students in Emm learn how to learn i how 
torplan learning activities, how to find and use resources in the 
coAwiLmity md how to build on es^erience. Learning becomes for them 
a /lifelong process with its own rewards directly related to each 
iridividual's personal choices and goals. 



THE (CElg PROGRAM 

since the fall of 1972, a model EBCE progr^n has been operating in 
Tigard, Oregon, imdar the sponsorship and technical assistance of 
the Northwest Regional Educatibnal Laboratory (NWHEL) and with 
fun^ng from the National Institute of Education (NIE) . The Tigard 
version of EBCE — ^called Commmity Experiences for Career Education 
or (CE) 2--is a f ull-tiTO educational altemative for youth in ttisir 
junior and senior high school years. The progran serves about 10 
percent of the eligible student body at Tigard High School. 

Ihe majority of student leain^ng takes place at sites in the southwest 
Portland metropolitan area, y^r^n f^t idents are not pursuing learning 
activities in the coimnuni^, ;r homa base is the (CE)2 learning 
canter. Staff at the learnim enter are not teachers in tiie 
traditional sense, but facilita Drs of student learning, helping / 

^student^ design and fariOT^^wii osfn -lamrntng^ plaii B w lth in ^^a 

prescribed curriculum and program completiOTi requirements* Volmiteers 
at community sites serve major si^port roles" in student learning. 
Policies for (CT)^ are detemined by a board of directors COTiposed 
of students, parente, employers, labor leaders and school district 
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rapresantativas. When students leave (CE)^ they receive a mique ^ 
portfolio displaying tiieir Gommunity experienGes and accomplishnients , 
and upon canpletion of program requireintarits they receiTO a standard 
diploma from Tigard High School. 



PILOT SITES 

After the NWREL EBCE progrMi was developed and refined over a three- 
year period at the (CE) demonstration site in Tigard, it was then 
tried out in four school districts in Uie Northwest* The EBCE 
progr^ was operated locally by each of these- districts with NTOEL 
providing trai.ning and tecjuiical assistance. Pilot sites were in 
operation ^.in Colville r Washingtoni Hillsboro , ^Oregoni Kennewick, 
Washingt^rfi ; and Kodiak, Alaska, during the 1975-76 school year. 
Experierices gained by the OTEEL EBCE avaluation unit from working 
with =the staff in the demonstration md pilot sites have served as 
the /basis for writing this grograin Evaluation han^ook. 
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THE EBCE HANDBOOKS 



Progran E valuation completes the five-volume set of handbooks which 
detail how to set up and operate an EBCE program. Built on its' 
ej^erienees with the Tigard EBCE model, NTOEL has comniled these 
handbooks after several years of development, evaluation and 
refinement of the program. 

Each of the EBCE handbooks is devoted to a particular area of 
program operations: Management S- Organization , Curriculum a 
InstructiQn, anployer/Comiuni^. Resources" / Student Services and 
Evaluation. A program overview brochure complem^ents the" handbooks 
and provides an introduction to EBCE. The brochure contains general 
information about the EBCE curriculum, key program elements and 
program evaluation. 

Users of the Progra m Evaluation handbo^.k will want to have the above 
materials on hand for reference to the more detailed explanations 
of program elements which they contain. The handbooks are summirized 
below. 



MANAGEMENT & ORGANIZATION 



MMafement & Organization treats ov^ ill operational considerations 
.or n-, EBCE proc .jui ouch a program Is organized, governed, 

star fed and made visible to tne public and how everyday program 
business is managed, ^e handbook is divided into four sections ^ 

"Program Planning s Governance" outlines' the steps for -^ tting 
up and operating an EBCE program, including conmunity' 
involvement in program planning, suggestions for surveying 
potential support, meeting legal and educational requirements, 
y m-uig program approval and providing an adequate base for 
policymaking. 

"Personnel" describes the general staff functions that are 
needed to operate an EBCE program. It also suggests procedures 
and considerations for recruiting and selecting staff, 
determining salaries and benefits, establishing working 
conditions, orienting staff and providing staff development.^ 

"Business Management" discusses such operational details as 
budgets. finatvUal reports, office procedures, insurance/ 
health and safety provisions, facilities and transportation. 
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"Conflnunity Relations" suggests strategies for' iiitroducing EBCE 
to the Gomiunity and meeting the ongoing information needs of 
various audiences, both inter^l and external, including board 
mmbers, staff, students, parents, business 'and labor, the 
educational conmunity and the coOTnunity-at^large. 



CURRICULUM a INSTRUCTION 



Curriculian & Instruction covers the content and processes of student 
learning in EBCE and the resources a comiunity-based program makes 
available to students. There are nine sections, to this handbook: 

"Curriculum Outcomes" ^provides basic definitions of key 
curriculimi elements and describes what students learn while in 
the program; the section includes general outcome goals and 
specific objectives to help staff prescribe for student 
performance in three broad areasi ^Life Skills, Basic skiiaB 
and Career Development. 

"Learning Plan Negotiation" focuses on techniques i 
i' Uvidualizing student goal setting and prescribing learning 
plans tailored to each student's needs, interests, abilities 
and learning style. Topics in this section include proq- >m 
requirements, accountability standards, assessment, learning 
site analysis and negotiation of learning activities. 

"Career Explorations" describes planning and impiementation 
steps for students' first experiences at workplaces in the 
local community* 

"Projects" describe ^ planning and implementation steps for . 
developing individualized learning contracts with each student 
that combine activities in Life Skills, Basic Skills and 
Career Development. 

"Learning s Skill Building Levels" describes p?anning and 
implementation steps for more extensive student involvement in 
projects and skill development activities at workplaces in the 
conmunity* " 

=» 

"Competencies" describes planning and implementation procedures 
for insuring that students acquire the essential survival 
skills needed to function in today's society. 

"Student Journals" describes planning and implementation 

----- procedures for encouraging .stiMent.use_ of ^Jaurnals^m 

of relrlectlng on personal experiences and building trust 
relationships with staff , 
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"^ployer Seminars" describes planning and implementation steps 
for utilising conmunity rapresentati\fes in large group student 
seminars on important career development topics and issues. 

"Learning Resources" describes ElCl's approach to using the 
community as a vast resource for student learning and details 
proced^es for finding, accessing and using learning resources. 



EMPLOYER/COMMUNITY RESOURCES 



Employer/ Community Resources treates the establishment, maintenance 
and use of the network of employer and coMiunity sites at which 
most student learning activities take place. This handbook 
consists of three sections: 

"Site Recruitment" details procedures for involving employers 
and other community site personnel in the EICE progrMi. Topics 
include the role and functions of the employer instructor, 
estimating the necessary nmnber and types of sites, incentives 
for participating in EBCE, identifying and contacting potential 
learning sites and adding sites to the network. 

- 

Employer Instructor Development" describes how participating 



site personnel are prepared for 



EBCE responsibilities. The 



section focuses on planning and conducting development sessions 
to give site personnel the information and training they need 
to work effectively with studehts. 

"Site Utilization" deals with the use of employer and conmunity 
volunteers and sites to deliver student learning. Included 
are procedures for assessing the educational potential of 
individual sites (learning site analysis) , supporting employer 
instructors as they work with students (site maintenance) and 
exchanging Information arong staff and between staff and site 
personnel, as well as staff responsibilities for working with 
site personnel. 



STUDENT SERVICES 



student Services covers Considerations and procedures for admitting 
students to the program, keeping records of student work, 
crt-dertialing students when they leave the program and supporting ' 
individual student growth. The handbook is divided into three 
sections I „ . _ , . 
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"Program Entry/Exit" details alternatives for enrolling students 
in ths EBCE program and preparing them for a new type of 
education. Topics discussed in this section include criteria 
and timelines for student recruitanent and selection, recruitment 
presMtations, selection procedures, orientation sessions, 
transfer into and out of the program and program completion, 

"Student Racords" covers procedures and considerations for 
collecting, recording, interpreting and reporting information 
on student progress through the EBCE program. Sample forms are 
displayed, including a student credential that provides a 
permanent record of student performance consistent with the 
individualized nature of EBCE. The section also includes a 
discussion of the confidentiality of student records - 

"Guidance" conce-ns those services, processes and interactions 
that help students understand and benefit from their individual 
experiences. This section includes discussion of program year 
action zones, the student accountaBility sy s tern, ^zone progress 
meetings, zone debrief ings and referral to outside agencies. 



PROGRAM EVALUATION 



"^^^ Program Evaluation handbook is designed for easy access to 
" how-to »do-lt" information. It contains two basic parts s 

ThB "Overview" sets the background for "understanding EBCE 
evaluatibn and how it relates to other elements of the program. 
A glossary of key evaluation and EBCE terms used in this handbook 
is provided* 

"Steps to Follow", organizes the evaluation procass into an 
easy-to-follow sequence of steps. The checklist which begins 
this section provides a useful guideline for the busy evaluator, 
administrator \or program staff member. ^ Each step in the 
checklist is keyed to a portion of the following narrative, 
which supplies details and discussion for each point. Cross-- 
references guid^ the reader to related material located 
elsewhere in the handbooks* 

Program Evaluation also includes Appendices of supplementary mterials 
and a reproducible materials section, presenting evaluation instrument 
which sctool districts may duplicate and use in evaluating their own 
EBCE programs. An index has been developed for this handbook to help 
users locate information quickly* 
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^Progran evaluation, &s an elOTient of EBCE^ consists of identifying 
evaluation puirposesi plajining an£ conducting an evaluation; reporting' 
and using evaluation findings. Evaluation focuses on both the 
planning and operation of the program as well as on program ou^^mes ' 

#^ ■ ' : ^ ^ 

Evaluation data are us^d at many .^levels and for making many 
deqislon^— feedback to students concerning their progress in various 
program activities, identifying specific areas in the program 
needing Impravement and determining program Impact, ' 

Evaluation data should not be gathered without a prior pur^se. To 
minimize interference with other program operations, only data 
essential to the total evaluation design should be collected. 



.WHY IS EVALUATION IMPORTANT? ' ■ 

ft. • 

EBCB evaluation serves liranediate progrMi needs for 

a* 'individual student baseline information which s;tudents and 
staff can use in developiftg learning programs tailored to 
each student . 

b, ongoing assessment data for refining individual rftudent 
goals ' 

c* feedback to students concerning their progress in various 
program activities \ 

i ' 

d.; identifying the apecific areas of the program needing 
improvement . . - 

-.-i" " . ■ 

Evaluation data serve the needs of external audiences (such as 
school boards, district officials and^ funding sotoces) .for 
determining the extent to which the prograin has met its' 
objectives^ assessing costs and analysing impact on the school 
syster^i and conmunity. v % 

pcternal audiences suchy4s federal and state funding agencies and 
regional laboratories also need accurate" and refined data 
concerning the effects/of EICE laa^ning strategies and program 
design.. Comparisons my be made, fa^ instance, with other types 
of career education ^ograms, Assessmfenf will also be made 
concerning the fidelity of EBCE programs to. their original models. 
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.Finally, evaluation results are likely to influence decisions 
qoncerning continued funding of EBCE* 



WHO'S INVOLVED. IN EB^CE EVALUATION? 

All adults md Etudents participating in an EBCE progrimt are to some 
degree involved in its evaluation. 

Program staff lisa assessment information on a daily basis in 
working, with students and reporting on their progress, 

Particpating cormnunity resoi^ce people provide frequent 
evaluative feedback conperning the performance of individuai 
students with wtom they work, 

EBCT) program directors and school admin Istr a tors responsible 
for insuring th^ program works according to its design require 
evaluation data effecting implementation policies, 

^ird p^ty (external) evaluators may be involved in designing 
- and administering evaluation instruments and assessing their 
results . ' 



WHO CAN USE THIS HANDBOOK? 



This handbook is not intended to train persons to become asserts in 
educational measurement and evaluation, teaining seminars,, 
lujiversity. coursework and intemships are available | for that purpose, 
Instead^ the handbook is useful in helping EBCE staff and persons 
already familiar with research and evaluation methodology apply 
their skills appropriately to specific elements of the EBCE model 
developed by NWREL, ■ 

Presently, those educators most directly concerned with IBCE 
evaluation will be staff and administrators associated with various 
sites that are wprking with the Northwest Regional Educatiohal 
Laboratory to implement NWREL's version of EBCE in their community* 

In short, the H^dbTOk is designed to assist in evaluating a 
complete EBCE ^rogram^ although portions are useful in evaluating 
the adoption of EBCE to an existing educational program and in 
evaluating^separate EBCE materials and processes that may be used . 
independ^'ntry by a school district. 
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^ GLOSSARY OF TERMS 

The terms below are defined expUcItly as used with the EBCE program 
and in this hmndbook. The same terms could have different meanings 
in a different context. 



Baseline data— information about students reflecting their status 
prior to their entering EBCE \ 

Basie Skllls^-atudent training or eKperience in reading, mathematicB, 
oral or v^itten Goramunications 

Career DevelQpnient- -currlcul\3m component that focuses on student 
eKamination of careers at specific employer sites r comparison of 
site experiences withrpersonal aptitudes^ values and aspirations 
and learning about problems and issues in today's working wrld 

(CE)5" see Coirounity Es^eriences for Career Iducation 

Coding" the process of transferring data from one form to another^ 
especially from test booklets to coding sheets suitable for 
keypunching - . 

Community Experiencea for Career Education (CE) ^— eKperimantal model 
of CTCE in Tigard, Oregon, designed-tQ test the idea that young ' 
people can receive a comprehensive education by learning, directly 
from adults in the conmunity - 

Comparison group" a group of students having similar characteristics 
as those, in EBCE who are tested at the same times as EBCE students 
to allow for a comparison of their growth; the comparison group may 
participate in an alternative career/vocational education program 
or simply represent students in a regular school program 

Competencies — survival skills considered to be essential for adult 
roles as citizen,, wage earner, consumer and learnen part of the 
EBCE Life Skills curriculum component 

Conf identialityr of data— the protection of information about individual 
p^sons involved in the evaluation so that their data are not ' 
released to unauthorized persona or Ufed in a way that could 
embarrass or harm them 

Control qroup" a groi^ of students who have applied to enter EBCE 
but were not randomly sampled from the pool of applicants to 

^^participate in the program; these S'^deri^s are tested at the aame 
time- as EBCE students to allow for a^ comparison of their growth 
with that of EICE students 
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Data knalYsis---"methods of processing and statistically analyEing 



data in order to show, relationships, make comparisons or provide 
descriptions '"" I 

Demonstratio n project- ^the (CDj project in Tigard, Oregon, where 
the NWREL version of IBCE was initially, developed, evaluated and 
refined. This site also serves a staff' training and /dissemination 
role and presently demonstrates the utilization of t^e ebce concept 

EBCE — see Eaqperience-Based Career Education 

anployer site— anv conununlty workplace agreeing to host students as 

they learn more about careera while meeting other learning 
..objectives 

»_ 

Evaluati on design- -the conditions and testing schedule under which 
the effects of EBCE are compared with competing or contrasting 
treatments; also referred to as research design 

Valuat ion instruments^ -tests and other measures used to collect or 
re^cord evaluation information 

Evaluation plan--a comprehensive plan for an evaluation that generally 
Includes the purposes and audiences for the evaluation; objectives 
and questions to be answered; evaluation strategies and instruments; 
procedures for data collection, analysis and reportihgi timelinei ' 
ai^ assignment of evaluation responsibilities 

Experience^Based Car eer Edubation (EBCE)= -an educational innovation . 
based on the idea that the community can serve as an alternative 
classroom for comprehensive secondary education by providing 
. "reality experiences" in addition to "book learning" 

Experimental qroup--the group of participants selected to receive 
the experimental treatment 

Experimen tal treatment- - the program which is being examined for 
, efficacy, in this docianent this treatment will be the EBCE program. 

Format---the structirre in which data are to; be coded and keypunched 
for computer analysis i 

.Formative evaluation- - the gathering and \repor ting of data for purposes 
of improving a program or its specif^ activities 

Handbooks— a set of five books in iShree-ring binders which detail 
how to set up and operate an EBCE program 

Life Skills— curricurum component that includes six 'areas of' 

emphasis I critical thinking, functional citizenship, personal/ / 

social development, science, creative development and the / 

competenGies / 
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Matrlx--a two-way classification of sets of items with one set 
going across the page formlny colmtms and another set going down 
the page forming rowsi the intersection of a specific row and 
column is called a cell 

Monitoring data- -data recorded periodically throughout the school 
year on student accomplishments such as the n\M^er of student 
projects completed each quarter 

National Institute of Education (NIE)~ -federal agency providing 
nationwide assistance and guidance to research and development 
activities In education! NIE contracts with NWREL to coordinate 
EBCE development,^ evaluation, dissemination and training 

NlE" see National Institute of Education 

Nort^est Regional Educational toboratory (tTOREL)- -a nonprofit 
research and development orgMization contracting with Nil to 
develop/ implement and evaluate the EBCE concept 

NWREL- -see Northwest Regional Educational Laboratory 

Part D, VEA—a; section of the Vocational Education Act under, which 
some school districts have obatained federal funds to implement 
an EBCE program 

PERT" See Program Evaluation Review Technique - v . 
PHS" see Protection of Htmatt Subjectp 

Pilot sites— EBCE projects located in school districts who have 
agreed to use the complate mmt IBCE program at their ovm 
eacpense and who receive some tectaical assietance and training 
from mmh 

Posttest~a testing session conducted at the end bt the data 
collecting sequence of an evaluation study after or near the end " 
of the treatment, such as at the end 'of an EBCE program year 

Preteat-^a testing session conducted at the beginning of the data 
collecting sequence of an evaluation study before or at the ' 
beginning of the treatment, such as at the start of an EBCE program 
year 

Program Evaluation Review Tectoigue (PERT) - -a planning system foir 
identifying key events in an operationV when the^-%ill begin and 
end, and their interrelationships . ' , 

Program ye ar action zones- ^divisions of "the program year into three=» 
to-five week periods to facilitate student scheduling and ^ 
completion of learning activities ? 
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Protection of Human Subjects (PHS) - -guidelines issued by the federal 
goverranent or other agencies to insure against an invasion of 
student privacy and to safeguard the information collected about 
people 

Ra^om sample— the selection of a small group from a larger group 
without bias; typically this is accomplished by assigning each 
person a number, then using a table of random numbers to select 
the small group 

Standardized instrument g—instrvanents that have been carefully 
constructed by professionals and are administered with standard 
directions under standard conditions with Itandard scoring 
teGhniques and that result in scores that have a norm base 

Student outcome qoal,'!^ — a set of IBCB rrtudant outcomes in Basic Skills, 
;;Lifa Skilly, Career Davelopment and bjoader outcomes resulting from 
EBCE eKperi^ntial learning ^ ^ 

Student pro j ects"individualized problem-centered learning Strategy 
that blend| activities in Basic Skill: w Life Skills and Career 
Developmental activiti«ss are usually carried out on employer and 
commm,lty learning sites 

\ ^ / 

Surnmatlve evaliiatl on— the ''Tathering and reporting of dita for the 
pMrpomm of making an overall assessment of a progrMi'or activity 

Thi r g p^rr ty evaluation— jL typt of es^ternal ;evaluation conducted by 
an eva^uator who is neither crnnecte'd with the implementation nor 
the dn^^elapift^int of the trea^tmos^t of interest 

United BzAiijA Officii of Edu wation rLfeoE)- -an a;grency-^ mt .^the., fedipr al...... . 

governmtsnt; r ?:?pon^ibl^i f ^: the implementation of many new educational 
progrtris lin^L i^ar*: DS xw., they have pro>fided money tp a. nmnber 
of dist^.'jrs to i^^CfM^t WkCZ. \ /' 

USOl— see United '^t^.^m ' education ' ' 

Variabl es—the chihgs of interest in a study; for EBCE ^these include 

such things as #ubn act's bbk, ethnic background, test scores, 
, prc^^i;ikr^ processes completed^ and attendance . 

VlA- ^the Vocational Education Act passed by Congrags which includes 
funds for the implementation of model career education programs - 
under Part D of the act , ? 



' 1 o 



STEPS 
JO FOLLOW 



STEPS to FOLLOW 



The Steps listed in this section of the handbook cover the 
major tasks involve& in conducting program evaluation. 
This checklist can serve as a useful way to insure that 
important steps in evaluation are not inadvertently 
overlooked. As readers scan the steps, they can determine 
for themselves whether they have a need to refer to the 
narrative which follows for 'additional inforMtion and 
help related to a particular step. While the steps are not 
always carried out in the fixed sequence in which they 
appear here, they often occur in approximately that order. 



t^iABLISHING POLICIES AND GUIDELINES 




Define and clarify the program goal^ and objectivep 
and their relationship to the prograip's processes' 



St important questions that the. program staff 
eel' should be answered by the evaluation 
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Agree on basic purposes and audience needs for 
evaluation 

Inspect local, state and federal guidelines as ; _ 

necessary to further clarify evaluation puiposes 

Review confidentiality of data and protection of 
hmnan subjects guidelines as necessary 

Agree on roles and responsibilities for evaluation IB 

Determine the budget, personnel mid resources 
available for the evaluation ' 



IDENTIFYING NEEDED INFORMATION 
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Determine what specific decisions are likely to be " 

made ^out the program md identify decision 25 

alternatives 
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Eit^lish a priority of what infOCTnation is' to be / 
collected in view of the resources available ^ 
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Deseribe the program and student populations to be 
evaluated . 

Identify data sources and appropriate ways to obtain 
the information needed 

Consider existing instruments that might be used " 

Identify and develop otiier instrmnents needed 
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by appropriate persons 
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CONDUCTING THE EVUUATION . 

Obtain student baseline and monitoring data 
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Analyze the data 
Report the evaluation findings 
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steps to FoUowl (Item 1) 



Establishing Policies and Guidelines 




Agree on basic purposes and audience needs for evaluat 



ion 



Basic purpc'ies and audience needs for program evaluation have been 
suminariMd in the overview section on pages 3 and 4. Below are ' 
examples of evaiuation purposes and some guidelines which should be 
considered in meeting those purposes and responding to various 
audiences. " 



INDIVIDUAL STUDENT PRETEST DATA 

The EBCE program has been designed to provide relevant evaluation 
data to various audiences both'Within and outside of the project 
Much of the data is needed on an Indiviaual,. student level while other 
data are needed at a group level. Sometimes the same piece of ' ' - ■ 
infoma is needed at both levels. For example, when students 

, enter EBCl they are usually administered a brief student application/ 
background questionnaire plus a series of evaluation instruments used 
for pretest purposes. Information about a particular student ' s career 
intorests, level of performance in various arias of the basic skills, 
and educational and occupational aspiratiohs is useful to the EBCE 
staff in developing 'learning programs tailored to each individual 
student. For example, career interest data identified through the 
Se^f-Direeted Search, the Career Informat ion Systaif or some, similar 
instriment provide staff with an understanding of the initial 
occupational areas students may wish to explore or which are suited 
to their interests aM/or competencies. Data from the Comprehensive 

'Test of Basic Skills or other achievement tests give an indication 
of which students may need additional diagnostic assessment, tutoring • 
or special work in programned instruction materials in reading or 
arithmetic. These Basic Skills data also help staff set criteria 
for ihitial student performance on projects. 



GROUP PRETEST DATA 



Dat^ collected on Individual student interests, vocational 
preferenQes, and, knowledge and performance levels in Basic Skills 




PROGRAM EVAWATION 



are also useful when reported at the group level. Such information 
allows the staff to understand the range of students ' ability in 
the program I to determine if particular strategies, such as tutors 
in Basio Skills, may be needed on a broad basis i and to identify 
program areas that may need special attention, such as providing 
eniployer-conducted seminars on techniques for having successful 
job interviews* These sa^e data serve as a baseline from which to 
measure the growth of students in program goal areas over the 
course of tho year. 



FORMATiyE EVALUATION 

The above discussion has illustrated how student pretest data can 
be used effectively at the individual student and group level to 
develop individualized student learning programst identify 
instructional areas needing special focus, and provide a baseline 
for measuring student growth*' A fourth purpose for evaluation, 
coiranonly called f oCTiative evaluation , is to identify ways in which 
the program may be improved* Questionnaires completed by students, 
staff,: parents and employers gather perceptions of the program's 
strengths and weaknesses and recoiranendations for. change. These 
questionnaires should be supplemented by informal daily contact among 
students, staff and employers that allows program participants to 
express their jud^ents of how things are 'progressing and to identify 
areas where chahge may be needed* These questionnaires and informal 
conversations can be' supplemented by other types of formative 
evaluation, such as interviewing a sample of employer instructors 
wlio attended an employer training session, to soliGit their reactions 
and to determine what they perceive as tlieir role in EBCl, 



ACCOUNTABrLITY 



A fifth purpose for evaluation differs from the others in that it 
addresses the concerns of an audlence^external to those participating, 
in ElCT—that is, district officials, school board members and the 
coiraunlty in general* It aims at; identifying the extent to which 
program objectives were met, how 1^ costs were incurred^ the 

impact on the school system and . coipunityV and any side eff bets the 
program may have, such km its possible impact on other career or 
youational education programs within the district* 
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PROGRAM ADOPTION 



people outside of the distridt may have information needs which your 
evaluation data could supply. Representatives from other districts 
considering the potential adoptiori of EBCE, or, in cases where 
external funds are used to operate the project, funding agencies 
such as the state or the United States Office of Education (^SOE) 
inay find interest in your results. Th^Bm external audiences are 
intarasted generally in the same issues as those listed under 
the discussion of accountability. In addition, they ^re likely 
to be interested in issues such as what problems were involved in. 
initiating that type of pro-am and how consistent a given IBCE 
progrMi's operations are with respect to the mmL model of EBCE, 
Evaluation can also assess any modifications that might have been 
made to the IBCE model and identify reasons for these modifications. 
In addition, funding agencies generally look for evidence to help 
them decide whether to continue funding of the prograin for the 
following year. 

Before developing a specific evaluation plan it is useful for the 
program evaluator ^d project staff to meet to identify the major 
pt^poses and audiences for the evaluation. Together they can 
systematically decide if the audiences will include students, staff, 
parents, participating employers and conttnunity resource people, 
school district personnel, sctool board members, funding agencies, 
other Interested educators and the general public. Such a decision 
influences 'steps 7 to 9 which deal with identifying information 
needed for the evaluation. 
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Inspect local, state and federal guidelines as neeessary 
to furtiier clarify evaluation purposes ~ I 



If a district is recaivlng state or federal flands to operate ain 
EBCE program, it is essential that toy evkluator and project j 
director be familiar with a^ evaluation guidelines \^prepared by 
the funding agency. School districts' receiving federal fiinds from 
toe U*S* Office of Education torbugh Part D of toe Vocational ! 
Education Act^ for example^ need to bi familiar wito toe Part D 
guideHnas which include esqpectations for a third party objective 
evaluation*^ 

Even if no outside funding is used to operate an VIBCB jprogram/ toe 
project director and person (s) conducting the evaluation shouldr 
check wito toe s^ool district to determine any local policies I 
related to program evaluation* Such policies might relate to \ 
parental approval for a young person's participation in toe 1 
evaluation; procedures for confidentiality of data, protection ok 
human subjects and review of evaluation instrisnents by toe ^ 
coranmiity; or use of school district resources in conducting toe I 
evaluation. . _ \ 



Exemplary ProjmctB in Voaatiohal EduGation, Critmrla for 
Selection of Applicants for FY 76. Federal Reglstar / Vol. 40, 
No^ 220, Nov* 13, 1976, 
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Review Gonfldentiality of data and protection of human 
si^jects gmdalines as nadessary 



ThB issue of protecting the rights of hiunan subjects has recaived 
Increasing attention in evaluation during the past five years* 
men developing new programs, ttera are six elements that should 
be reviewed to inside tiiat neither i^e program nor its evaluation 
methods cause harm to progrm participants i 

1* program design 

2. evaluation .design 

3» evaluation instruments 

4, security of data 

5. confidantiali^ of individual resi^ndents 

6* release of findings* ' . 



In reviewing the design of a pro- am such as EBCE, it is in^ortant 
to insure that it does not cause students to close options in their 
lives. For eKample^ it would be unfortunate if a program were to 
look students into learning only^ ^out their first occupational 
choice and to ignore otiier possibilities throughput the year. * 
Likewise, it would be haraful if EBCE provided for the needs of • 
; students planning to enter trades after graduation but ignored 
the preparation of students who might want to attend college * 

In developing an evaluation design the desire for a thorough 
evaluation should be balanced by a concern for safeguarding student 
and staff time, ^us, while eight hours of pretesting might give 
an evaluator "comprehensive" data, such a procedure i^ores the 
rights of the students, iu^ extensive testing also may cause an 
undue negative attitude toward the program and may result in the 
colleotidn of invalid data. Sensitivity to coriserving student 
time, for exan^ple, has led, EBCE b valuators to recDiranand actoinistration 
of only three si^tests from tiie Comprehenpiva Test of Basi^c Skills 
rather than using toe entire batte^, thi reducing test time for 
Basic Skills from fbur hours to one hour* 

Evaluatio n instr^ents should be essential, related to areas needing 
assessment, aid should not contain items tiiat could cause a student 
erabarrassTOnt or tiiat are strictly personal.. To insure toat 
instrtanente used by ttam in evaluating EBCS are appropriate, the 
NWBEL EBm evaluation unit has had their own instruments reviewed 
in advance >y an independent PTOtection of Hman Si^jects Committee 
at NWREL," 
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It is also useful to have a-distriGt*s EBGE Advisory Committee 
review and approve evaluation instruments in advance to insure 
tl-mfei the items are not offensive to the local cciranunity. As an 
\ addicional precaution, individual students should be free to omit 
an item, si±>test or test if they feel it would be harinful in 
some way 

After evaluation data are collected, it is impDrtant that they be 
kept in secure place s until thmy are keypunched, scored or no 
longer needed. This prevents unauthorized persons from having 
access to student scores or other results. IBCE staff should also 
keep any confidential notes on individual students in secure places 
sudi as locked file cabinets, - 

men interviewing parsons or asking them to complete questionnaires, 
you should guarantee tiiem that their individual responses will be 
kept confidential and tliat only a group suranary of the da^a will 
be released, This is particularly important when respondejtes are 
asked to identily program probla™ or weaknesses. In cases\^here 
the e valuator receives data on individual students ttrough the 
mail from districts , student code nuntoers can be used and later 
matched wito proper names* Data received should be maintained 
securely in lockable metal file cabinets, j 

A district planning to contract for the evaluation of Ef^should 
have a policy regarding how and by whoni findings will bs used and 
releafed # 

Guard against ttie inappropriate release of evaluation data to the 
media. Standardized test scores for each school should not be 
released without proper adjustment for or inclusion of sociOGcoriomic 
or other pertinent data. Misleading data of tois kind have caused 
eie piiDlic to form inaccurate^ interpretations and to demand changes 
that may not have been warranted* 
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Agree on roles and responsibilities for evaluation 



Agyjpments about who v/ill plan^ nonduct and report EBCE Evaluation 
re^-t^ts should be reached by a schncl district shortly after the 
decision is made to conduct an evaluation. Depending on the basic 
purposes for an evaluation/ the evaluation might .be conducted 
informally by the project director and staff or more formally by 
persons witoin or outsid?^ of the school district. Guidelinea from 
funding agencies sometimas specify the type of evaluation to be 
conducted* 

EBCE sites receiving USOE^ Part D, VEA funds are required to have 
a third party evaluation performed by a person or agency external 
to the district and the Mdel of EBCT being utilized. Sites not 
receiving Part D fu.ids arta free to determine who will conduct 
their evaluation. Since evaluation is built into liie EBOT 
instructional system, students and project staff are necessarily 
involved in an internal evaluation* To the extent that a district 
may desire credibility for the evaluation results , an external 
evaluation can be very beneficial* However, even when an external 
evaluation is performed, close cooperation with the internal 
project staff is essential. 

Several additional factors are important to consider in deciding 
who should conduct tile evaluation* One of these is the availability 
and cape^ility of an evaluation unit wltiiin tte district itself* 
Some districts have a hi#ily trained and efficient research and 
evaluation staff that is es^rienced in conducting various types 
of evaluation, pmen tdiis is toe case, tte district may wish to 
have the evaluation done by that unit if it has the available time 
and resources. Ilie district may also wish to inwlve an outside 
person or agency to work witt che district in such areas as setting 
up an EBCl evaluation design, designing any special instrments that 
my be needed, scoring and analysing data, or maintaining an up-to- 
date file of ^e status of all program participants. 

In tJie case where a district does not haw a skilled evaluator , * - 
a^^ilable within the district and funds are awilable, it may \ 
wish to contract for all or part of the evaluation* When selecting 
an eval^tion individual or agency, the following factors should 
be cons ider ed i 

1, training of toe evaluator in research and evaluation 
methodology 

2, ai^erlence in evaluating educational prngrait^ 
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3. skills in developing new instrimients tiiat may be needed 

4* data analysis capabilities 

5* familiarity v?ith career or vocational education 

6. avail^ility and geographic proximity to the EBCE site 

7. skill in relating to students ^ employers and public 
school staff ^ 

Districts can save substantial amounts of money by having their 
own staff actually collect moat of the data. Wien performing 
this XQ1&, staff should understand tte purpose for each instrmnent 
and should have had some training in how to administer it* 
(Escplicit step^by-step instructions for administering selected 
instruments are given in ^pendix 1,) 
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Datemina the budget^ personnel and resources available 
for thm evaluation 



Before preparing an evaluation plan, it is useful to estimate the 
budget available for conducting toe evaluation and to identic 
personnel and resources that could be made available for evaluating 
the EBm program, Thm resources available (or not available) often 
influence whether an evaluation is oonducted by project staff 
evaluators within the district or external evaluators. Available 
evaluation resources witoin tiie district can also affect the tasks 
left to be parformed by an external avaluator* For example, if 
an external evaluation is being contracted and the local district 
maintains an efficient print shop, it is probably most cost- 
effective for the external evaluator to prepare a camera^-ready 
copy of instriraients and reports for tJie district's printing or 
duplicating facilities • 

In Gonsidering personnel within tiie district who could assist in 
the evaluation of EBCT, it is often useful to look beyond persons 
trained in research and evaluation* For example, an EBCT program 
may be able to arrange for toe district's director of curriculmn 
and instruction to evaluate a a^ple of student written projects 
and reports using a predesigned rating sheet. Teacher aides could 
be trained to collect certain kinds of data* 
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Ideniifying Needed Information 



Define and clari^ the program goals and obSectives and 
their relationship to the program's processes 



Goals identify what accomplishments are es^eGted by ttia EBCE 
program. They usually deal witt student learning outcomes but can 
also be directed to other processes or outcomes of a program ^ 
such as the active participation of coOTnunity resource parsons or 
the recruiting of at least two employer sites for each student in 
the program. Program objectives specify in greater detail what 
is to be accomplished and provide evaluation criteria for measuring 
discrepancies between what is plamied and what actually occurs . 
A very generalized objective may be a valuable one in cbncept, but 
difficult to evaluate* Before evaluation can proceed, it is 
necessary to review the program's goals and objectives to be sure 
tiiey are es^ressed in clearly undergtood and measur^le language. 

With individualized programs^ such as EBCl/ not only can student 
learning processes vary for different students but student 
objectives thejnselves can also va^ among students,, Thm mmh 
version of EBCE contains sonu. outcome objectives toat are identical 
for all students (suc^ as toe competencies)^ some Uiat apply to 
all students at a general level but vary in the way tiiey are defined 
for Individuals (such as projects in each Life Skills area) , and 
some outcomes that are co^letely optional for students (such as 
in-»depth study of a particular career tiirough^ a skill-building 
level) , 

At the demonstration site in Tigard/ Oregon^ NWREL e valuators and 
^ project staff developed 22 student learning outcomes covering the 
program areas of Basic Skills^ Life Skills, Career Development 
and broader maturation goals* The mmh evaluation unit foimd it J 
helpful to keep these student outcomes at an intermediate level 
of detail rather than to speci:fy tiiem in strictly b^avioral teme. 
Individually negotiated student projects, on Uie other hand^ 
contain specific b^a^orally-^stated objectives for a given student 
together with toe resources toat student has agreed to use and 
the criteria that will be used by toe learning manager and student 
in evaluating those objectives* 

Local EBCE project staff need to work wito the e valuators in 
defining and clari^ing major student outcomes. In doing so it 
is useful to keep in mind that certain goals (such as improved 
competence in arithmetic) can be considered direct outcomes of the 
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program and activities developed with each student to lead to those 
outcomes while other goals fsuch as openness to change) are indirect 
outcomes and may not have specific activities geared to achieving 
tiiem. 



RELATING LEARNING PROCESSES AND OUTCOMES 



A matrix can be used to clari^ the relationship beta^een student 
learninc processes and desired student outcomes. In the sample 
matrix which appears on the following page major outcomes are 
listed along one dimension and learnirig processes along the second 
dimension. Such a matrix is useful in establishing that one - 
outcome may be achieved tiirough more than a single process. It 
can also help establish the extent to which EBCE is an individualized 
progrMi wi^ optional paths avail^le for different students. It 
is useful for the project staff to work with the evaluator in 
developing such a matrix before evaluating a particular EBCE 
progr^. This type of matrix has also b^en found useful by project 
staff in clarifying expected outcomes for each learning process 
used in a project* ^ 



CROSS --REFERENCES — 



Curriaulum a Instruation (Cai) han^ook aontalrw^..ir^e tailed 
description of student competenaims (pages Life Skills 

projectB (pages 195~198) , and thm sklll-bulMf^ IbvbI (paaes 277- 
278), ^ - 



to mKamplm of an indivlduall^^Qtiated student project is found 
on pages 200^204 of Currim^mi & Instruction hanmook. 



EKLC 
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" EBCE PROCESSES 



^ot^U This is 3 mampl^ 
ma t ri >f ; tertnk a PP^^ ^i ng 
hQro r^fl&Qt but >ire not 
i den titj&I wi th t^t ms 
used to d&scribe the 
current NWEEL model ,o£ 
EXE, 




Career Kno wi ed* 



XX Primary Proaesm for mQhIeving a student Qutcome 
X Secona^ry Process for aahieving a student outcome 

Each learning process is described fully in the Curricul 



j./ jgcruetion handbook. 
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Detamine what specific decisions are likely to be made 
about the program and identic decision alternatives 



As a preparation for identi^ing information needed for the 
evaluation, it is useful to clari% project objectives and their 
relationship to the program's processes as described in step 6. A 
second source for identi^ing information needed for the evaluation 
is to determine what decisions . are likely to be made about tiie 
project and to arrange to collect data useful in making ttiese 
decisions. The timing of 1±ese program decisions can also be of 
critical importance where refunding or possible es^ansion are involved. 
For example / the e valuator in interviewing tiie project director 
may learn that the project is linking of coniDining l^e role of 
learning manager witii that of employer relations specialist. If 
tiie project had a total staff of seven^ the evaluator might help 
the staff set up an evaluation of staffing patterns in which 
several staff contjined the two roles within the same person while 
others continued to function in one or anotiier of tiie separate 
roles. The reactions of employers^ students and staff to these 
Wo alternative staffing patterns could be evaluated togetoer with 
other criteria such as tiie average lengi^ of time it took in each 
of the two situations for a student project to be written or 
reviewed and the quali^ of resulting projects. 
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List important^ questions tiiat the prograin staff feel 
should be answered by the evaluation 



During the planning phase and first year of operating; an EBCE 
project the evaluation can often be most useful by focusing on 
important process or outcome questions that ^e program staff feel 
should be answered by the evaluation, ^ These questions usually 
go beyond asking whether t±iB project's objectives were achieved 
and sometimes deal wiui issues not addressed by tiie objedtives of 
the project, l^ie box below shows some of tiie formative evaluation 
questions raised and addressed in the first few years of evaluating 
the EBCE demonstration project in Tigard, Oregon. 



EXAMPLES OF FOWATIVE EVALUATION QUESTIONS 

1. How effective were the EBCE student reGruitment 
procedures and^ how could they be improved? 

2. How is student descriptive information conununicated 
to and used by eE^loyer instructors? 

3* How many occupational clusters are represented by the 
EBCT- participating employers? 

4. ' Is the Student Accomitability System understood by 

students and f^ctioning effectively in terms of 
utilization ^ follow- through and results? 

5, How much involvement do students have in setting goals 
for their own learning and for selecting content and 
designing activities to achieve these goals? 

6* i^at influencr- if any, has EBCE had on the District's 
Cooperative Work E^^erience Pragram or existing 
vocational education programs? 



A list of important formative evaluation questions can often be 
developed during the suimar or at the beginning of a school year* 
However, an effective foimative evaluation should also be responsivi 
to special information needs that may arise from ^e project staff 
at any tima during tjie year* In many cases the evaluator can 
suggest to the staff ways in which tiiey can collect and analyze 
data that are important to them in operating or modi^ing ^e 
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program. A particular concern regarding fonnative evaluation is 
l^e need for timely reporting of inforaation. For example, 
the results of a simple questionnaire conpleted by part±cipants 
at an EBCE employer training seminar and a discussion of staff 
reaGtions to tiie seminar may be most appropriate on tiie day 
following the training seminar. Such information is important i] 
improving future seminars - 
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Establish a priority of what information is to be 
collectad in view of tiie resources available 



After Gompleting steps 6 thro,Ugh 8 it is quite prob^le that tiie 
information identified as nee,d^d for the evaluation (based on the 
list of project objectives^, anticipated decisions to be made, and 
formative evaluation questions) is far greater than can be collected 
witt limited evaluation resources* Priorities must be est^lished. 

It may not be essential or even appropriate to evaluate each of the 
identified studerit outcomes during a particular year. In general^ 
it is often more desirable to concentrate on evaluating process 
objectives during the first year or partial year of a project's 
operation than to concentrate on student outcome objectives^ 

After project objectives have been clearly written and agreed upon 
by the evaluator and staff/ the evaluator should try to gain a 
sense of relative importance among the goals in the opinion of 
the staff. This can be achieved by asking the staff individually 
to rai^ order each student outcome or at least to indicate opinions 
as to whether each outcome is of hi^, Mdium or low importance in 
their EBCE project* A tabulation of these independent ratings and 
group discussion can often lead to a general agreement within the 
group about which outcomes should receive the major attention 
during toe course of a year. 

The evaluator mi^t also work wilJi project staff and high school 
officials to determine whether each student outcome could be 
considered wiique to IBCT students ^ .whether it is shared by 
students in other prograM within the district (such as a Cooperative 
Work E3^erience Program) , or whetiier it is cono^n to all high school 
students witoin tiie district* This kind of distinction is important 
for determining whether an outcome should be measured Only for 
EBCE students or whether it might also be valid to measure it for 
students in another alternative progrMi or for students in the 
general hi^i school population, Suc^ decisions are important not 
only for determining how each outcome to be measured (see steps 
11 to 13 on pages 33 to 42) , but also for deciding what conparison 
groups might be relevant to use with a particular student outcome 
objective (see step 10 on pages 30 and 31) , 

Itiare are a few criteria that may be useful to project staff in 
helping to establish a priority of infoCTiation needs * Hhmmm 
criteria have been presented in question form and appear in the 
box on the ne^t page. 
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CRITERIA FOR ESTABLISHING A PRIORITY 
AMONG EVALUATION INFORMATION NEEDS 



1. is lAe evaluation of t^ls objactive or program area 
required by federal, state or local guidelines? 

2. Is this objective or area central to tte success of 
the project? 

3. Will toe information gathered be of in^rtance to 
project decision^makers? 

4. Has this or similar information alrea^ been collected? 
(If yes, do we need to address toe same issue again?) 

5. Has the project been in operation a sufficient length 
of time to warrant evaluating this area at this time? 

6* Do we have (or can we obtain) sufficient expertise to 
adequately evaluate tois area? 
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Preparing an Evaluation Plan 



Describe the program and student pMulations to be 
evaluatad 



A comprehensive evaluation of ebCE should include a description of 
the program characteristics and the stt^ent populations involved. 
Our evaluatioa e^^erience with the EBCE pilot sites has suggeatad 
that a description of certain context characteristics is useful in 
COTmunicating an understanding of the ^setting. These characteristics 
includei the commujiity, the district, the local high school (s) 
from which EBCT students come and local career/vocational education 
alternatives available for students* 

A description of the local EBCT) program itself would generally 
include objectives, organizational structure, {including staffing 
pattern), managCTient procedures, curricul™ components, learning 
processes, student recruitment ai^ selection procedures and costs. 



STUDENT POPULATIONS 

In addition to describing .the local EBCE program it is also important 
to describe the appropriate st^ent populations* TOese students 
include those in EBCl itself, those in any comparison or control 
groups and perhaps a random sample of^4tudents from the regular 
high school. Part of the student group descriptions may be obtained 
from cumulative file data from prior years such as achievement test 
results (if available), prior grade^^int averages and average 
number of days absent from sctool. 

Use of various types of student comparison groups can provide an 
important additional dimension to the evaluation of certain sti^ent 
outCOTies for which an external comparison is valid. For e^cample, 
in assessing the growth in career knowledge aM attitudes made by 
IBCE students it wuld be appropriate to compare such gains with 
those made by simile students who volunteer for a specific school- 
based career education or work study program* Without the use of-- 
sueh comparison groups it becomes im^ssible to address the issue 
of whether the EBCT students might not have been able to make the 
same gains over the school year without being in the program. 
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Three types of comparison groups are useful j 1) a random sample 
of studants from the regular high sehool, 2) a control group of 
students who originally volunteered for EBCE but who were not 
randomly selected to participate and 3) students in a school-based 
career/ vocational education or cooperative work experience progrM, 
Each group is discussed below. 

Random Sample 

The random sMiple of students at the high school can be used to 
determine the extent to which students who voliinteered for the EBCE 
program were similar to or different from the "average" high schowl 
student. This infoCT^tion is important in understanding the nature 
of students attracted to EBCe. Selected information such as aptitude 
or achievMent data already on file for all of these students from 
the past few years can be used as the basis for this comparison 
tc^ether with the St^udent Background Questionnaire th%t can be 
completed by the random sample and comparison students in September 
and October, This questionnaire contains a subset of the items, 
whiqh are administered to the EBCE students, dealing with past work 
es^eriencer attitudes toward school and parentis education and 
occupation. Except for tois questionnaire, che r»dom sample 
students would not need to ba tested or surveyed again* 

Control Group 

A control group consists of tliose students who Initially applied 
for admission to the EBCE progrmi but were randomly selected not 
to participate. (This group could exist and be used only if t±e 
pool of student applicants for EBCE exceeds the number that can 
actually be atoitted by at laaat 20 and a random assignment of 
students to EBCE is made from all those who apply.) Tlurough randOTi 
assignment, students in the true control group can be assumed to be 
of the same background and ability as those admitted into EBCE, 
Therefora, gains made by these control group students over the year 
in areas such as Basic Skills can be separated out from tte gains made 
by students in EBCE to get a picture of the gains that can be 
attributed to the treatment effect of being in EBCE, 

Comparison Prograni 

The third comparison group is that of students in a school-based 
career/vocational education or v^rk experience program* Because 
these programs may share some of the same objectives as those of 
raCE, students in this group can be tasted at the beginning and 
end of the school year on those instrmients that measure the coiranon 
objectives of the two programs* Tiie results of this comparison 
will provide evidence regarding the extent to which the outcomes 
produced by the tWQ programs differ* 

i 
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AVAaABLE INSTRUMENTS 

f 

Three Instrianents have been developad by the mmt IBCE evaluation 
staff to assist in describing thi pr ogr an and > student population^, 
. to be evaluatea--the EBCT Essential ''aar^cteristics Chaeklist, the 
EBCEProoeis Cheekliat and the Student A^plication/Backgrourid 
Questionnaire, A- copy of each instruinent is looated in the 
Reproduaible Mata^iais seotion of this hanibook* 

The EBCB Essential Characteriitics Checklist was designed to 
describe, the extent to, which a project-poasesses the philosophical 
foundations of a NWi^L EBCB site. It covers five component areas 
; of the prograin. These components are that EBCli 1) ia an 
individualized program, 2) is a comnunity-based prograini 3) is an 
experience-based progran and is built from the career activities 
of adults^ 4) must have its own identity and be comprehensive and 
integratedf and 5) places a major emphasis on thi career 
development gt students. Each component area con tain%, from four 
to six essential ch«acteiistics • Kiese charlcteristiis are rated 
on a scale of 1 to 5 v^th the anchor points prespecified. 

A second checKllst^ the EBCl Process Checklist, is intended. to 
. complement trie EBCB isseritial Cteracteristics Checklist by 
identifying operating procedittes used in an EBCB pr^ram. This 
profcess checklist contains four sections i 1) BBQi objectives^ 
2) management and organisation processes? 3) curriculmn and 
instruction processes and 4) student services processes, 

A third descriptive instriaiient is the EBCl Student App'lication/ 
Background Quastionnaire, When stude^^s formally apply to enter 
IBffi they are asked to complete a brief application questionnaire 
that records information about family background, student's previous 
employment history^ short-" and long-range educational and work 
goals I past involvement in high school and commiuiity activities 
and reasons for wanting to enter the EBCE program. This information 
is used to describe the students entering EBCE and comparison groups 
Changes over-the year in M^e^^guch^atf students ' eduoationai and 
occupational aspira^jjons-crn^be documented since some of the same 
questions appeaF on both the Student Application/Background 
Questionnaire and on the Student End-of -Year Questionnaire, 




steps to Follow (Item 11} 



Identify data sources and appropriata ways to obtain the 
infonnation needed 



Once an agreement has been reached on the areas to be evaluated and 
on the priorities among these areas, it is then essential to 
identify data sources and to decide on the most appropriate ways to 
obtain the information needed, 

Thm data sources include both the persons and docments from which 
evaluation infomation can be obtained. E^^erience gained from 
evaluating the NTOEL EBCE demonstration and pilot sites indicates 
a number of useful sources for obtaining important evaluation 
information* Thm bosc on the next page displays these sources and 
examples of data tliat can be collected from each soiirce* 

EVALUATION STRATEGIES 



A compel ehensive dlscuasicn of evaluation techniques and strategies 
is beyond the scope of this handbook. However, there are a nianber 
of pr£:ctical resources available in this area.l Because time and 



The reader may wish to rm&d the section on "data sources" Im 

Andmrson^ S.S,^ Ball, S. ^ Murphy^ M^T^ and ABsoaiktmB^ 
Enaydlopedla of Educmtional Svmluation , Smn FrmhaiBao: 
JoBBey-^BaBB^^ Inc., 1975, 

Sections of the following may alBo bm of Intermit: 



IsaMOf S. and Miahaal, W.B. Handbook in Ee&Barah and Evaluation , 
San Diego f Ca,i Robert Knapp Co., 1971. ^ 

C^onB, T^, Maeim^J^^ and Fehrer^aaher H. "Thm Ubb of Multiple 
S\:rmtegieB in Evaluating an E.'Tperimnae^BaBmd Career Education 
Program^" Re&earch, Evaluation and Develo^ent Paper B^rieB 
No. 9. Northwemt Regional BducMtionMl Laboratory, 1976, 

Worthen, and. Sanders , J.R. Eduaational Evaluations 

Theory and Practice . Wor^hington^ Ohioi Charles A, JoneB 
PubliBhingCo.,1973. 
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f 



Participating 
EBCE students 



Control and 
cpmparlaon 
group students 

Graduates of EBCE 



Dropo^ats from 
IBCE 

Parents of 

EBCE students 



BnplQyers or 
conmunity 
resourae 
peopla 

EBCE staff 



Project 
dir actors 



Student 
completed 
projects 



USEFUL EBCE DATA SOURCES 

B^mmplB^ of Data 

Student's scores on measures of basic 
skills I career developmenti and attitudes 
toward^ the progran^ themselves^ education 
and work 

Scores of measures of basic skills i career 
develojpnent; attitudes toward themselves, 
education and work 

Hieir current educational or occupational 
activities, de^ee of sati . faction with 
their activities, and reflections on the 
usefulness of the IBCE experiences 

^eir reasons for leaving EBCE and their 
attitudes toward the program, 

aeir descriptions of the strengths and 
weaknesses of EBCl, benefits of the 
progron to their son or daughter and their 
involvCTtent in the program. 

Their judgment of the progress of students ' 
with whom they worked, the impact of EBCE 
within their organization and their 
perceptions of the program's operation and 
their role in it 

llieir judpftent regarding the importance and 
effectiveness of various EBCE learning 
strategies, factors contributing to and 
factors limiting the success of the program, 
and reconmiendations for program change 

Wieir perception of the degree to which 
their site has implemented the mmL version 
of EBCEi description of their site, the ^ 
local school district and the community; and 
program cost data 

Judgments about the extent to which the life 
skills projects met their goals, capitalized 
on student interests, were experiential in 
nature, and effectively utilized community 
sites 



steps to Follow (Item 11) 
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evaluation resQUrce.^ are important constraints, it is generally wise 
to consider a variety of potential evaluation stragegies and then to 
analyse the alternatives to select the; minimal number of gtrategies 
ne^fedsd to accomplish the job. Pactorsj important to. consider In / 
detaMiining priorities among potential j stragegies include i 1) co^^ts, 
2) timing^ 3) credibility of findings,! 4) degree of obtrusivenesi^ 
5) toount of coordination needed, and 6) efficiency* / 

Cost ^ i . , ^ / 

Cost factors to be considered include the cost of purchasing or 
developing an instrument as well as tl^e cost to administer, score 
and analyEe it, Thm amount of time i^nvolved of students, staff 
or conroimity resource persons is of^n overlooked but should be 
taken into consideration in determining costs* 

/ 

Timing / 

The timing of an evaluation Strategy is another crucial factor. 
Several elements of timing need to be considered including? 1) the 
deadline when the information is needed, 2) the length of time it 
would take to plan, collect and analyze data, and 3) the most 
appropriate time in the developmental cycle of a project for 
collecting certain data* 



Credlbnity ^ . 

Credibilily of the findings is something tlmt is often overlooked 
until an evaluation is completed and the data reported* Some 
audiences suoh as educational researchers may be impressed with 
"hM-d data" such as a multivariate analysis of scores from a 
itandardiged test. Other audiences^ such as parents, may be more 
informed by well written student case studies* In addition^ 
credibility of the evaluation findings can be enhanced when several 
evaluation strategies produce results that reconfirm or support 
what was found through the use of a single strategy* 



Degree of Obtrusiveness 

Another important factor to consider in selecting evaluation 
strategies is the extent to which a given strategy, will be obtrusive 
and perhaps clash with the mission and activities of the project* 
The evalvaation of a sample of already completed student written 
projec^.s provides an exeellent insight into student, learning 
outcomes without requiring additional time of students* On the 
other hand, it may be unreasonable and inappropriate to administer 
a three or four hour basic skills test battery twice a year if any 
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of the students in that particular program joined because of a ' 
dislike for classrooin-oriented activitias. 

Coordination . 

A point oftan ovtelooked in selecting appropriate ways to obtain 
needed information is the amount and type of coordination required. 
This coordination includes ,the amount and type of interference that, 
may accrue to Jtudents, staff, partioipating employers and others. 
It also includes the coordination of persons or agencies' to be 
involved in the data collection for each strategy. . One compromise 
the authors Have used with the operations staff at the demonstration 
site xn managing the coordination of different evaluation strategies 
was the agreement at the beginning of the school year as to the 
maximum total amount of direct time of students, and staff that would 
be used in any type of evaluation. This agreement insured that the 
project's coordinator of research and evaluation would maintain 
close planning and monitoring of student and staff time used in 
evaluation. , 



Efficiency 

Although we have mentioned that credibility is enhanced when several 
evaluation approaches related to the same issue produce consisten<- 
findings, one has to weigh the Relative value of redundancy of 
information collected on a few issues to establish the reliability 
of the findings versus loss from foregoing the collection of unique 
information about additional issi 



MEASURING STUDENt OKCOMES 




One way of organizing and displaying the evaluation techniques to 
be used in evaluating EBCI outcomes is to prepare a matrix 
portraying learning outcomes and various evaluation strategies 
that will be used in assessing these outcomes. An example of s uah 
a matrix is shown on the following page. Sturfent outcomes are 
listipd among one dimension and evaluation instruments or sources 
of data are listed along the other dimension. This display 
demonstrates that some evaluation measures cover several outcome 
-areas and conversely that a single outcome can be assessed by more 
than a single measure, such a display is helpful also at report 
writing time since the evaluator can use it to relate the findings 
of various measures in reporting findings lor a particular outcome. 
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EVALUATION MEASURES 
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Consider existing instruments that might be used 



Two areas of IBCE putcqines where existing instrtiments may be useful 
are in measuring Basic Skills and Career Developnent. The Life 
Skills area of EBCE oontains some objectives for which few if any 
standardised inatrmients exist. 



ADVANTAGES AND DISADVANTAGES 



The use of standardised instr^ents in evaluating BBCl has potential 
advantages and disadvantages that need to be weighed. by an ©valuator 
and project staff. Some of the potential advantages are that 
standardized inatrumants usually havei 

1* established research supporting their va],idity rfnd 
reliability , ' 

2, notTTB that ailow comparison of the perforniance of students 
with others around the eoufltrj( .ja^t ^he same grade level 

3, Qiirric^Iuin--free oonterit (thus making the instrument equally 
fair-T^or unfair^-for IBCE and cbroparieon group students) 

4, general credibility with school board members and 
administtitors who may be important audiences for the 
evaluation report . 



Some of the potential disadvantages of Using standardised insttiunents 
are thats ^ t . 

1. some of the outcomes they measure may be less relevant to 
EBCB students and to real world performance needs 

2* they fail to identify specific competencies that students 
have or have not achieved ■ - : 

3. they are often less sensitive to program feffects ov^r a 
short period of time (i*e*^ within a single school year) 

4. the validity of their Results is destroyed by changing 
individual test items or administration procedures 

5. they may not be based on a norm group which is comparable 
to the students served by the project 
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steps to Follow (Iteni 12) 



\ CONSULT REFERENCES , ' ' 

B^^^e selectifi^ a particular atandardized instrument for use in 
mck, the tvalua,tor may wish to read the review of the inatrraient 
in a general referenoe source (e,g,, Burosl) , The review of 
standMdi^ed instrments done by Research for Better Schools and 
the UCLA Center for the Study of Evaluation may be uaeful in 
deciding such factors as examinee appropriatenese, normed excellence, 
teacher feedback, usability^ and retest potential, ^ Many 
standM^dized career developnent inatrumenta have been evaluated by ) 
a panel of measurement everts in career education. 



j SPECIAL CONSIDERATIONS IN SELECTING EXISTING INSTRUMENTS ^ 
Has tha Instrument bean used before in evaluating an EBCE Project? 

If a particulM' test has been used in EBCE by otif of the foi^ 
regional laboratociea responsible for development of EBCl models 
you might contaot ^he l^oratory involved to determine the rationale 
for ita selection, problems noted in its use or interpretation^ 
,and the nature of the evaluation findings. 

What InstrumantSs If anyi is the school district already using with 
high school students? i 

If a district is already atoinistering (and plans to oontinue using 
in the future) a standardised Baslo Skills aGhievement test to all 
high school students at the grade levels from which EBCE draws ^ 
you should oonaider thm advantages of using that same inatrt^ent- 
If this instrmnent is administered near pretest or posttest time ^ 
this would worm easily allow for oomj^fcing results of EBCE and 
non-EBCT students. It would also be useful for students who might 
transfer into or but of EBCE dicing the eourse of the year. 

\ . I . ^ ^ ' . 



\ 



2 



^^Burom, O, K, T he Seveiith Merital MmmBurBment YearBook , Highland 
Park^ N. J.^ Thm Gryphon Frass, 1971, 

\ . ' \ ■ _ ^■ 

Ho&pfnmrp R, et al. CSE^Rm Tmst Evalmtlonsi Tmmtm of Higher'^ 
Order Co gnitivB^ Aff&ativm and IntmrpmMorml Skills , Center far 
thm Study of EvmltiatiQn^ UCLA Ca, 1972* 

3 ' ■ . , ' 

Yomg^ M, fl. and Sahuh^ g. Evaluation and Eduamtional Decimion 
MaMm ^ Wamhington, DLd Pevil^ 
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Are the assumptions underlying the instrUrpent consistent with the 
EBCE philosophy? \ 

An- important consideration in selecting certkin eKisting measures 
of career knowledge and attitudes is whether \he assumptions 
underlying the instrument are consistent with EBCE philosophy 
and practice* For example^ some existing careers. maturity . . 
instruments have been developed for in-school programs rather than 
for experience-based programs operating in the co^unlty and thus 
assimie that students should get much of their vpca&nal information 
from the guidance counselor rather than from actuallk exploring 
careers in the community. Other assmiptlons behind eMsting career 
matitt^ity instruments are that students should have a, general 
knowledge about a variety of careers whereas the l^RIL Version of 
EBCE stresses; the importance of students having a knowledge of 
careers that are of particular interest to them and match\heir 
attitudes and competencies* \ 

- - - ■ . - I 

Is the instrument appropriate to students In EBCE? 

It is also import^t to consider whether an existing instrument is 
appropriate for the. reading level and maturity of the students for 
whom it is beijig considered. For exMiple^ the Self- Directed Search 
has been a problem for many poor readers in IBCl and for SOTie 
students who have little Jmowledge of many of the job titles 
mentioned in the inatrument* ^ 



Can selected subtjests of the Instrument be used to give valid 
Information while saving student time by not administering the 
entire battery? 

'\ ■ 
Boim standardised achievement^ teat batteries require over three or 
four houra to ccanplete. To cpnaerve student testing time, the 
asseasment of EBCE student growth in Baalc Skills can be achieved 
by administering only selectedl subtests of a complete battery. 
For example r in using the Comprehensive Teat of Baaic Skills , it 
ia possible to obtain useful growth meaaures related to EBCE by 
selecting, only the Reading Comprehension , Arithmetic Concepts and 
Arithmetic Applications si^tests. 

Is the instrument sex- or race-biased? 

Producers of standardized teats have become very aensitive within 
the paat few years to charges of a ex and race biaa in their 
insteumenta, it ia important in eonaidering a standardised test 
that the reviewer determine that the instrument is free of bias. 
This my be done by reading the actual items on t^e test and by 
reading the publisher's technical report "to see^\diat steps have 




Stmpm to Follow (Item 12) 



been taken In tiie developnent of toe instrument to iniure that its 
use and the interpretation of its results will not be biased or 
stereotyped, - 



yor n intereBtlng discmslon on iB&ues of smsc bims the reader mmy 
wish to read Is&mM of Bex BimB and s&x FairnmmB in Carmer intB^mst 
Memsurement edited by Emther Dimmnd^ Career Education Progrmnf 
Nmtionml Inmtitute of Educmtipn^ Spring^ 19^5, 
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Identify and develop otbar Instruments needed 



Stmdardized tests are not available for measuring most EBCB^student 
outeontes, OMs leaves the evaluator with, a choice of using existing 
n©n-standardlied 'instrOT\ents or developing, local instrumehtation. 
The Reproducible MaterirfJs section of this handbook contains 
evaluation instruments ^ --^eloped by NWOTL that address many of the 
.NWRiL EBCE Student outcomes. A rationale and description of each 
instrtfflient is also given in that section, Evaluators of mmh EBCE 
projects are free to 'reproduce and use any or all of these . ^ 
instrifflienta that fit their needs* 

These NVTOlL-developed instruments are based on EBCE student ©r 
progrMi outcomes and reflect ej^eriences shared by the other three 
laboratories involved in developing IBCE, by the danonstration site 
in Tigard and by. the NWREL, pilot sites, Kiey have been reviewed and 
approved by the mmm^ Protection of Human Subjects comnittee and by 
the Office of Management and Budget in Washington, D,c, Their use 
.•will permit a local EBCT project to compare its results with those 
obtained from other districts using >he N^ffiL version of IBCE, 
Districts wishing to use these instrinnents may contact Nram for 
further dnf ormation on- their use and on future revisions that may 
be made- Subh districts will be req^sted fumisb NP^L with a 
GOpy„af thsir results, ^ / 

To the extent that districts adapt the EBCE model to their local 
needs, new objectives or learning processes may be added which 
require additional ev^aluatlon instruments or modifications to those 
developed by NWOTL. hs m% -icample, most sites have added or 
st^stituted some new. stud?. * competencies for which valid 
certification procedures are neaded. Others have used various 
IndlvidualiEed Basic Skills learning packages which already have 
or require tlieir competenpy-based tests. 

Thm developnent of valid and reli^le non-standardlged Instruments 
requires technical knowledge of instrument develofwient procedures 
which can often be obtained through the .use of shprt-term consultant^ 
in measurement. Such consultants need to work cooperatively wito 
project staff who possess a good understanding of ^SBCE ^to Insure 
that any Instrimient developed la consistent with the IBCl philosophy 
and is appropriate for the intended respondents. 





Stmps to Follow (Itmn 14) 



Dev#iop componenta of a eomprehensiva avaluation plan 



Succasatul completion of the praceding items in this gection and 
ttosa in the previous section will lay an excellent fouMation for 
preparing an IBCl avaluatiqh plan. Although an evaluation plan is 
often prepared f6r only one year at a time^ it is useful in the 
first plM to include some general overview of the type of - 
evaluation planned for the next several years of the pro-am. In 
general^ the first year of evaluation involves heavy emphasis on 
formative (process) evaluation with less attention given to a 
smranative (outcome) evaluation. During the second and future years 
of EBCE (as the program begins to st^ilize)/ less emphasis is 
placed on forrative evaluation and more on guninative evaluation. 

In building a housed the preparation of # bluej '^Int OQcurs after 
preliminMy disGvsaions regardihg costs and the general needs that 
the house is to serve ^ but before toe actual construction of the 
foundation and walls. In a similar way^ the evaluation plan for 
EBCE should be prepared after preliminary policies have been 
established but before the data CDlleption has begun. 



ELEMENTS OF A COMPREHENSIVE EVALUATION PLAN 

' 1 * Purposes ^d audiences for the evaluation 

2, Project objective^ \ 

3, Questions to be answered by the evaluation 

i 

4, Evaluation strategies to be used 

5, Instruments to be obtained or developed 

6* Aasigranent of evaluation roles and responsibilities , 
between the evaluator and the project staff 

.7, Procediffes for data collection, analysis and reporting 

8, Timeline specification of what will be done by whom 
and when 



-PROGRAM EVkWATION 



Eight elements generally tomA in a Gomprehensive evaluation plan 
are shown in the box on the preceding page. The first six elements 
have already been discussed in this handbook* Procedures for data 
collection, analysis and reporting are covered in steps 16 to 23. 
A timeline , can be developed as shovm in the box below. 



1* Determine what 
to evaluate 

2. Develop evaluation 
plan and ' 
inetrtraents 

3 . Secure approval * 
of Instruments 

4. Collect pretest 
data 



. Analyze pretest 
data 

Collect process 
data 

, Collect posttest 
data 

8, Analyze process 
and product 
data 

9» Prepare final 
report 

10* Prepare report 
suranary 




5.; 
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If a more detailed work plan is desired the eyaluator may wish t© 
develop a PERT (Proc'ram Evaluation and Review Technique) network 
as shown in the box hslow. An advantage of PERT is that it makes 
more clear the rel .ticMhip among tasks, and identifies which tasks 
are dependent on ti-? ccTnipletion of a prior task and which tasks can 
be done aoncurrently . A task displayed on the same horizontal line 
can be accomr iished only when the task to its iiittnediate l^ft has 
been performed. Dotted ^inas are used to show coordination of 
aetivities that^are not ^.^cessarily dependent on eaw..' other. 
Cirelas indicate the SGheduled completion date of each actiyity. 
The boxes indicate reports or information, fee^ack. 



SAMPLE PERT NETWORK 




a Tasks In the Ne 



Decide on the nature of 
the evaluation 

Select internal or 
external evaluators 

Determine what j.a to be 
evaluated , 

Develop an evaluation 
plan and instEimem.s 

Gb% the plan and 
instruments reviei^d 



6. CQllect baseline data 
?• Collect oretest data 



8* Collect and analyse process 
data" 

9. Analyze baseline data 

10. taalyzs pretest data 

11* Prepare interim report 

12. Collect posttest ^d survey 
data 

13* Analyse posttest and survey 
data 



14. 



Prepare final report 



15. Prepare report summary 
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15 



Subniit the evaluation plan for review and ratification 
by appropriate persons 



Since the preparation of^a comprehensive evaluation design is the 
single most critical aspect of a good evaluation, it is worth 
allowing adequate time for 'its preparation. It is also useful to 
insure that the draft of the design is reviewed by the project 
staff, the project advisory coimittee,, the distriot: liaison person, 
and (if funds allow) by another evaluation specialist who could be 
asked to critique i^.. Once approved, the design can then be 
implemented, keeping in mind that unforseen changes still may 
crente a need to revise it at a future time , 

he NW-UEL EiCE evaluators have found that although the Project 
Director may be the only staff membjer who carefully reads the 
evaluation plan it is a useful idea for the evaluator or Project 
Director to review the general elements of the plan with the entire 
staff. This practice allows the staff to acquire an awareness of 
what is to be involved in the evaluation and of their participation 
in it- 



\ 
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Conducting the Evaluation 



In Bteps 1 through 15 we have discussed the planning^ phase of an 
EBCE progx^m ^jyaluation. Proamdu^es for actually conduatlng an * 
EBCE evjluation are aontainmd^in gt&ps 16 to 22, 



Obtain student baseline and monitoring data 



In addition to student data collected as pretest and posttest 
information it is im^rtant to obtain student baseline and monitoring 
data. Baseline data/ or information aTOut students prior to their 
entry into EiCE, is useful in describing the extent to which EBCE 
student applicants represent a cross range of the total high school 
population from which the prograin draws, ^ 

Information already on file^ such as aptitude or achievement test 
data administered to all high school students during the past few 
years can be obtained for students in EBCl and compared to a random 
sample |(or to the total population) of high schiDol students from 
which ^E^CE volunteers came* The grade point average (GPA) and 
nuit^r of days absent for the school year prior to entering EBCE 
can also be obtained for students in the prograin. In addition to 
providing a baukground description of the EBCE student population, 
the prior yearns attendance data can be used to compare with the i 
attendance data of the same students dicing their first year in 
the progrOTi, ^is comparison assmnes student attendance i^ 

an indirect indicator of student attitude tqward participating in 
EBCT, - 

Even before EBCE pretesting occurs on students beginning a program 
year. Initial student monitoring data should be collected. Student 
monituring data refer to descriptive data kjDOUt eaqh EBCT student 
and a record of hi i or her attendance and siJ^cific kccomplishments 
in EBCE throughout the school year, A list roster of prospective 
students along with their sex land grade level Status should be 
prepared* (See Student Services handbook, ^gei 7-26*) This list 
forms the bssis for recording and monitoring si^s%uent student progress* 

At recruitment or pretest time students complete an application 
questionnaire which provides additional student background data 
such as birthdate, reasons for entering the EBCE progrMi and baseline 
data on occupational interests and es^erianoe* Additlohpl 
information will be obtained for local internal use only ^ such as 
address, telephone nmnber and a parental agreement for the'xstudent 's 
participation in the program and its evaluation activities, \ 
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other baseline infortnation is callected in thm curricula areas of 
Basic Skills, Life Skills and Career Develop ,ent. As the year 
progresses a record is kept of student progress in such areas as 
numbers of explorations, learning levels, skill building activities, 
projects and competenGies completed per student, and tiie numLer of 
days absent. In addition, the type or name of each project and 
rompetency should be reported, 

Dioring the course of the year new students will enter the program 
and need to be added to the roster along with their appropriate 
background, testing and aGcomplishment information. Students will 
also le^Ve the program, either voluntarily or due to non- 
participation. Their information should be deleted from the 
evaluation record ^since they will have incomplete test data) and 
their names removed from the roster. 



CROSS-REFERENCE 



The gmthorlngr of baseline assBSBment data for usb in dBsigning 
individual studmnt learning plans is d&tMiled in JtOT J of "Learning 
Plan Negotiation" in the Curriaulma s Instruction handbook, pages 



^6 
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Plan and administer prates'ts 



WHEN TO PRETEST 

Instriiments usea to estimata growth in specific EBCE araas should 
be administered on two or mqre occasions— preferably as students 
begin EBCE (pretesting) and again near the, end of each year in the 
program (posttesting) . 

An ideal time to administer the pretest is shortly before the 
students actually begin interacting with the formal elements of the 
program. Since students are eacpected to bacome immersed almost 
iiranediately in EBCE, an ideal pretest time is about three weeks 
before program start-up for a new sohdol year. Several sites have 
pretested in August so they could gain information on students ' 
career interests and Basic Skills performance level for use when 
students begin program activities in September. Because all students 
might not be able to complete their testing at this time due to work, 
vacations or travel, make-up sessions should be planned to- occur 
shortly after the beginning of the school year, 

A smaom alternative is to test all students soon after the pro-am 
begins. This, however, can create several problems. First, full- 
scaled testing during opening weeks can unduly complicate the 
program's calendar, which, will already be quite full with orientation 
and other start-up activities. Second. EBCE is an alternative to 
the regular school program and its uniqueness might appear diluted 
by starting the term with standardized testing similar to that in a 
regular high school program. Finally, but most importantly, EBCE 
IS an individualized program that attempts to match student abilities 
interests and attitudes with projects and career explorations, it 
requires a thorough student assessment in the areas of Basic Skills, 
Life Skills and Career Development. Although this assessment should 
be ongoing, it Is critical that it exist quite early in the program. 
Delays in pretest administration plus the time necessary for scoring 
and reporting of data will delay full program implementation. For 
example. If pretesting is done during the first week of the program, 
data coding is done during the second week, and several weeks are ' 
allowed for scoring and reporting of information, staff and student 
use of individual assessment data might be delayed until the fifth 
week of the programs ■ 



A third alternative is to pretest all IBCl applicants in the late 
spring prior to their entry into the program the following fall 
This alternative has the advantage of allowing adequate time for 
scoring and processing student scores prior to active student 
involvement in EBCE, 
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Potential drawbacks to this alternative are that- some students, 
over the siffimier^ may change their minds about entering or not 
entering EBCE and thus some new students may still need to be tested 
in ^e fall, A second drawback is that it then becomes difficult 
to separate out the major EBCE progrM effects from those resulting 
from esQseriences such as suiraner jobs. 



POINTS TO KEEP IN MIND WHEN TESTING 
Explain your Purposes 

Be sure to tell students why you are testing and encourage them to 
do their best. The primary .purposes for EBCE pretesting are— 
1) baseline data are necessary so that individual project 
negotiation, employer site placement and student counseling can be 
accomplished properly, and 2) student growth needs to be assessed 
to see how well the project is achieving its objectives, 

Ehvlronment and Other Consldirations 

Test a<Mninistration should be imiform for all groups of students 
tested. Directions for administering each of the evaluation 
inatrmnents should be studied by the testers, prior to the day of 
testing since these may affect tha sir.e of the group to be tested 
at one time, room conditions or t^ ^ jed,*f^ spe,cial ^at^ - ' 

(such as stop watch, if a timmd is bdlng^usedj/ 

The testing location should be^well lighted and have adequate 
ventilaticn* ^ere should be sufficient testing space for each 
student, both for comfort and to minimize the possibilities of 
copying* Different levels of the sane test may be needed if prior 
data indicate a very wide range of student abilities, fflie timing 
of tasting sessions, where appropriate, should be strictly followed, 
EeLch student should haw a sharpened No, 2 lead pencil which is 
required when Digitek or other machine--read^le TOSwer shaets are 
used* It is a good idea to provide these shaa^ened pencils and 
collect them with the t^st booklet and answer sheets* Extra 
pencils should be available for use if needed. 

Careful roonitorlng during the testing period is necessary, especially 
when special answer sheets are being used* It is important that 
students understand how and where to mark answers since only portions 
of son^ answer sheets are used and there are wide format differences 
among answer sheets,! During this tune the examiner can also check 
to see toat the correct pencils are being used since marks mad^ in 
ink are not readable by machine* 
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When students coinplete the tests, examine them imn«'/.iately for 
missing iections, multiple responses and lack of identification. 
In this >fay the difficult problem of going back to the students' 
to match up or correct answar sheets can be avoided, 

Follow-Up 

Immediately after student testing it" is useful fur the test 
administrator to wite down any irregularities that have occurred 
during the testing period, ;^is infornation is , useful in helping 
to interpret unusual results. Scoring and coding of evaluation 
instriMents should begin as oon as possible after the instruments 
are administered. This minimizes loss of data and insures quicv .r 
availability of pretest results. 



— — — CRpSS^REFERENCES ^ — . 

Specific directions for administering NWREL^dBvelopBd instruments 
are given in Appendix ■ 
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Collect process and survey data 



"Prpcass data" refers to data that can be collected periodically 
■ thrdugriout the year to monitor the direction of the IBCE program 
and the progress being achieved by students* "Stirvsy data" consists 
of interviews or questionnaires that have been developed to assess 
the opinions students, parents, staff or comunity people 
associated with EBCE* 

Two instr\snents have been developed by the NWREL EBCE evaluation 
staff to assess the program processes being utilized and to 
detemihe the degree of fidelity of the replicated progrMt to the 
NWREL EBCE model: the NWREL EBCE Process Checklist and the mroSL 
EBCE Essential Characteristics Qiecklist . 

i 

These instruments were developed by NWREL to^ — 

1. provide descriptive information to NWraL staff on local 
pr jgram characteristics and the degree of fidelity of the 
Ic cal model to the NWREL EBCE model ^ 

^ 2, estimate program drift in essential program features and 
processes over a period of time 

Local project staff can use the results of these two instruments 
to— 

1. gain fee^ack on the extent to which they are implementing 
the EBCE model 

2. identify any operational areas they may wish to change in 
order to meet original expectations 

3* docmnent local deviations from the EBCE model 

Both instrT^ents should be completed by the project director and 
evaluator from two to four months into the program year after the 
progran has had a chance to stabilize. Administration at an earlier 
date would result in information more subject to change as the 
iroplCTentation adapts to local constraints or demands. Later 
administration would aelay providing feedback to the local staff 
concerning discrepancies .between their model and Uie NWREL EBCE model. 



brimf dmsaription mud a copy of each instx^ument is located in 
the Reproducible Materials Section of this handbook. 
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Both instriiments can again be adi^dnistered about month or two 
bafore the end of the year to assess the mount, if any, of program 
drift during the year when compared witt the first atoinistration 
results. Such administration should occur prior to pDsttesting and 
tefore the end-of-the-year push begins. 

Student prograss data such as the number of GOmpatencies, projects, 
joi^nal entries, es^lorations and learning levels completed by each 
student are generally collected by the operations staff at the end 
of ea^ of eight to ten sehocl year action zones toroughout the year. 
The program evaluator may wish to summrise the progress of the 
total group of EBCE students on completing these activities on a 
zone=by=zone basis or at the end of each semester. 

The NWREL EBCE evaluation staff have found it useful to collect 
survey questionnaire data from parents and conmiunity members 
associated with EBCE aroimd March or April of each year * This 
timing allows a period for follow-up reminders to members of these , 
groups if the initial response rates are low. it also decreases 
the nimber of activities needing to be completed at the very end 
of the school year. It has been useful to collect questionnaire 
data from students and staff during the last month of the school 
year since many students become even more active in the program 
during the last part of the school year. 



CROSS-REFERENCES 



The uses for the studmnt progress or monitoring d&ta mre dBsaribed 
on page 64 of this handbook. ' 
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Plan and administer posttests 



Tha tarm "poattest" im used here to refer to evaluation instriisients 
adinlnistered to students after some specified period of time during 
which thiey participate in EBCE. Because student participation in 
EBCE can be designed to last from one semester to several years it 
may not always occur at the end of a program year , although that^ 
occ\^s most frequently. Since students tend to become restless 
during the last few weeks of the school year and many are TOrking 
extra hard to complete nroject assigrments necessary for graduation 
or end-of-year promotion, it is reeoirmiendHd that students be tested 
approKimately tliree to foiur weeks before ^e end of the program 
year* This will also provide tinte for make^ testing sessions 
where necessary* 

All growth-related instrmnentation administered at pretest time 
should be administered again at the end of the year to evaluate 
changes that occm-red over the course of a student's participation 
in the program. In additdon/ use this posttest time to collect 
data which would have been inappropriate to measure at pretest 
time (e,g,^ student attitudes toward the program and staff/ future 
plans or hindsight comparisons with the traditional high school 
programs) . End-of-year questionnaire data are essential to^ collect 
from both students and staff* Bierefore, this posttesting session 
is likely to take slightly more time than did the pretest. 

When administering the posttes't all of the precautions indicated 
under pretesting (step 17) should be repeated. It should not be 
assmed that a test administrator will remMber all of the procedure 
to follow or i^at students will understand the purposes for the 
testing* 
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Score and/or code data 



The evaluator of an mCE project needs to decide -whether project 
data are to be scored and/or coded within the district or contractad 
to a person or agency outside of the district. If the data are 
scored or coded within the district a decision is needed as to 
whether it will be done Mnually or by machine. 

The decision wheth« to contract with a person or agency outside the 
district for scoring and coding services is influenced by factors 
such as? 

• The availability of trained personnel within the district 
to provide these services 

• The availability of necessary mchine scoring facilities 
within thb district / * 

• l^m availability of funds for an outside contract 

• The costs of either service 

• accuraoy likely under either ser\^ice 

A decision as to whether instrunents should be scored manually or 
by machine is influenced by factors such as? 

• TOe nimber of instrmients to be scored 

• The availability of trained personnel^ equipment and 
programs for scoring the instru^ients 

• ^e turn-around tMe of either system 

• The costs of either syatmi i 
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Analyze the data 



Thm analysis of EiCE data can be a time consiaiing activity* As was 
true undar step 19, the evaluator of an IBCE project needs to 
decide whether project data are to be analyzed within or outside of 
the district and whether or not the analysis will be done by 
computer. Thm same factors irifluencing the decisions in step 19 
are applicable here also. 

Discussions of specific statistical analysis is beyond the scops of 
this handbook* V instrimients administered as pre- and posttests have 
generally been analyzed by the NWREL EBCE evaluation staff using 
a multivariate analysis of covarience, with the posttest scores 
serving as dependent outCOTie variables and pretest scores as 
covariates, The MULTIVARIANCE package by j". Finn has been foimd 
very appropriate for such analysis. 

Survey instrmients such as the Student End-of-Year Questionnaire, 
Parent pinion Survey, Employer pinion Survey ard Staff 
Questionnaire have been analyzed at NTOEL using the CROSSTABS 
program from the Statistical Package for the Social Sciences (SPSS) . 
Thim program is convenient for analyzing data acrcss several groups 
such as for first year and second year program participants. 

In deciding on the types of analysis to use. It Is Important to keep 
in mind the level of the audience for whom the evaluation reports 
will be addressed. 



Finn, J. MULTIVARIANCB: A generalized univariate mnd multivarimtiB 
analyaiB of variance , covmrimncef and regression Vers ion 
Chicmgoi NationBl Educational JReBouraes, Inc., Marah, 1972. 

Nie, N. et. al. SPSS Statistical Packa^a for the Social Sciences . 
Second Edition, New York: McGraw-Hill Book Co., 1975. 
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f 

Re^rt the avaluation findings 



In reporting evaluation findings it is useful to review what 
questions each audienGa hds about the topic to be reported. By 
keeping the audierice in mind, the evaluator can better datermina 
when to give the report, the level of presentation and appropriate 
methods of reporting. 



Timing 

The tlBiing of evaluation information reporting is important for both 
oral and written reports. In EBO: the re^rting to students and 
parents occurs frequently and systematically throughout the year. 
Use of the school year action zone provides feedback on the level 
of progress expected of each student by thm end of specified 
calendar periods. Other eMmples of timely feedback to students 
occur when the learning manager meets with a student to review a 
recently completed project or when a cOOTnunity mraber certifies 
whether or not a student has successfully demonstrated completion ' 
of a specific competency. 

At the program level, timely feedback can occur infomally when a 
staff man4>ar calls a participating employer prOTiptly to discuss 
a problem mantionad by a student while at that site, whan an 
evaluator provides the draft of a special report to the staff which 
will be used in planning for the next Mployer SCTiinar^ or when an 
evaluator discusses an interim evaluation report with staff for 
their use while preparing a continuation proposal for the following 
year* Interim reports are also useful to funding agencies if the 
decision regarding the continued funding of a pro j act must be made 
prior to the availability of a completed end-of-year evaluation 
report, - 



Level of Specificity 

The question of how detailed or specific to make a report also 
relates to an understanding of an audience* If the report is to 
be given about an individual student, both parents and student^s 
are^ usually interested in specifics. If the report contains data 
for an entire group, it may be best simply to summarize it and 
place the details in an appendix, ^e saihe data may be reported ^ 
to various audiences at various levels of specificity. For mKsmple, 
a complate evaluation report n^y describe the instruments used to 
measure each particular objective, the student population or sample, 
procedures used to analyze ths data, the statistical findings and 
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a discussion of the findings related to each objective. Project' 
staff may be interested primarily in the findings and a discussion 
of how they could be used to improve the program. Parents or 
cQnrounity persons may be interested in a slagle sentence smranarizing 
how students performed on a particular o): lective. 

Reporting Methods ^ ^ 

Evaluation findings may be reported by one or several mettiods 
including oral^ written, or multi -media presentations. For 
example, a report to a school board may be a complete written report • 
However, if only 20 minutes of teard time were allowed for a 
presentation, the evaluator might wish to summariEe the findings 
orally and show some tables or graphs on an overhead projector that 
highlight the main findings. The evaluator may v/ish to share soma 
recommendations orally with the toard in order to get their 
reactions. 



THIRD PARTY EVALUATION REPORTING 



Third party evaluators, or those contracted from outside of a 
difitrict to conduct, an independent evaluation of an EBCE project, 
should be guided by the needs of the project staff, the school 
district and those of the funding agency, p^ile the needs of the 
project staff and school districts will vary from district to 
district, the guideJines from all funding sources (including USOE 
Part D, Vocational Iducat ion Act^) should be reviewed carefully. 
Guidelines written specifically for judging the "third party 
objective evaluation" part of the proposals me&ti also appropriate 
to consider when recalling information to be reported, 

A helpful practice used by s"ome third party evaluators is to give 
an informal oral report to the project staff at one or more times 
during the course of the year and to give them a draft of the end 
of the year evaluation report in time to react to it and challenge 
any factual errors or omissions. This latter procass allows the 
evaluator to make any factual corrections in the final version of 
the report before it gets sutoitted to the district or funding 
agency. The operations staff may still not agree with the 



GuidelineB for those receiving Part D funds are avuilablei 
Exemplary Projects in Vocational Education, Criteria for Selection 
of Applicmnts for FY 1976. Federal Register , Vol. 40, No. 220, 
November 13, 1976, pages 53962-S2965. 
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interpretation of the findings or recommendations made by the 
eyaluator but there should be agreement on the factual data* 

Some evaluators allow the operations staff to write an appendix 
section to the report in which the staff summarize their 
interpretations of the findings and the^r rdcoranendations in cases 
where they differ from those of the evaluator. This process 
maintains the objectivity of the evaluator while allowing the 
district and the funding agency to read a different interpretation 
of the findings when such differences occur. 



CROSS-REFERENCE 



Pages 66-69 of the EBJE Currlclim & Instruction handhook describe 
haw the school yemr actloji zones operated ^ 
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Using Evaluation Findings 

■■ ^ . "v:-: ■ _ ; L " - 

In steps 16 to 22 we have dimcuBBed taBkB involved in conducting a 
program evaluation. Procedures for using evaluatidn findings are 
contained In steps 23 to 25, ~ ; . 




Use student pretest data 



Student pretest data can hm useful at both the individual student ^ 
level and at a ^jroup sunmary level. In 1976-77 the following 
backgromid und pretest data were collected by the EBCE pilot sites i 

1* student file data including recent achievement test data^ j 
GPA and attendance records from the ^ear prior -to entering/ 
EBCE \ " / 

2, responses from the: EBCE student Application/BaGkground / 
Questionnaire j 

3» results from the reading comprehension and arithmetic 
subtests of the Comprehensive Test of Basic Skills J 

A, Student Attitude Questionnaire responses 

5, EBCE Seman;tlc Differential responses 

Some sites are also using the Self Directed Search as a pretest, 



STUDENT FILE DATA 



At the individual student level, the student file data can give the 
EBCE project staff an indication of each student's academic 
achiavement and general school performance as ,^ef lected in the GPA 
and attendance. This knowledge is Useful ^ for example, when the 
learning manager begins to plan student projects with a student* 
The attendance rate is also useful as a comparison measure with 
student attendance while in EBCl* 
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^STUDENT APPLICATION/BACKGROUND QUESTIONNAIRE 



Th^ EBCE Student Applieation/BaGk^oimd Questionnaire contains much 
infotTnation that can be useful to SBCE staff in understanding a 
particular student. Information is provided in areas such as family 
background, reasons for entering EBCE, the student *s previous 
employinent history, short- ar^ long-range educational and work goals 
recent involvement in high school and conmunity activities , hobbies , 
and reading habits,, Some of these areas such as student's 
educational and work goals are^repeated on the Student Ind-of-Year 
Questionnaire as a means of determining changes over the year. 



COMPREHENSIVE TEST OF BASIC SKILLS (CTBS) 

The reading comprehansion , arlthmatic applications and conGepts 
subtests of the eras are administered by many EBCE projects as pre- 
and posttest ihstriattents to measure student growth in Basic Skills. 
Project staff have found the pretest scores useful in determining 
which students may need diagnostie testing to determine specific 
areas needing intensive work in Basic Skills and in judging the 
level of Basic Skills activities to be incorporated into students' 
individually negotiated projects. 
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STUDENT ATTITUDE QUESTIONNAIRE 

The Student Attitude Questionnaire is being used for the first time 
xn EBCE during the 1976-77 school year. This instrument dontains 
Items intended to reflect short teim' growth on the part of EBCE 
students tlmt will exceed growth made by comparison or control 
cfroup students. This instrument gpntains four sections! the New 
Mexioo Career Oriented Activities ^b^cklist developed by Educational 
Evaluation Associates, the School OpiMon Scales dewloped by Par 
West Laboratory, and two sections- dexre loped by NWREL— a Life; Skills 
Attitudes Scale and a Conpetencles Scale. Thm New Mexico Career 
Oriented Activities checklist is part of the Mew Mexico Career 
Education Test Series. On tiiis 25 -item subtest students select 
responses that reflect career oriented activities. For exampie, 
students check whether, during tiie past year, they "found out the 
occupations people wiUi interests like mine enter" or "learned 
which training programs. are open to me and which ones interest me " 
The next two subtests contain items with a five-point rating scali 
ranging from "strongly ^gree" to "strongly disagree." items assess 
attitudes toward decision =maklng, adiool, peers, . staff and the Life 
Skills areas. Thm final scale contains a number of competencies and 
the student checks. yes or no as to whether he or she has already \ 
ccnnpleted a particular eertipetepoy. >, 
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Pretest results from^this instrument can be especially usGful to 
project staff in counsaling students who have done few or no career 
related activities in the past and those who have a particularly 
negative attitude toward peers ^ staff or the Life Skills areas. 



SEMANTIC DIFFERENTIAL 



The Semantic Differential measurek student^ attitudes toward the-. 
Qoncapts of school ^ adults ^ learning and work with each concept 
rated 0|i a five-point scale for 15 polar adjectives. Thus eich 
concept can have a score of from 15 to 75, Project staff may wish 
to identify and counsel with students having la particularlir negative 
attitude toward any one of these areas on th^ pretest. 



THE SELF-DIRECTED SEARCH 



Although the Self-Directed Search , developed by John Holland, has 
not proven particularly useful in EBCB as an evaluative instrument 
for measuring student growth over, a one-year period in career v 
development, it can have value to program staff as a pretest 
instrument, - 

Self-Dlrebted Search (SDS) provides toe student! and staff with 
a picture of the career areas that are of high or low interest to 
a particular student. ^ Such information is often used by IBCE staff 
in selecting some job sites for that student's initial career 
explorations, A consistency and differentiation score ajre also 
provided for eapl)- student, Thm consistency score is a Measure of 
the correspondence of a student's pMmary and' secondary occupational 
preferences with Holland's Theory of Careers, Thm differentiation 
score reveals the degree to which a student can dif f arentiate N 



1 

For a further dBscription of the six SDS oaaupation&l/perBonality 
aodmB devmlopmd by Holland and the consistency and differentiation 
saor&B of the SDS s em pages €-6 and 7 6f the Appendix to the FY 75 
Final Bvaluatlon Report of 'the NWREL ExperlBnae-Based Career 
Education Program. For further discussion of help an EBCE staff 
membBr may be able to give a student with a low score in these 
V areas, please see John Holland's paperbaak entitled Making 
' Voc&tionml Choic&B: A Theory of C areers (Prenticm-Hall / Inc ^ 
' 1973)1 ^ ~^ - ' 
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between his or her high and low interests, k particularly low 
score on any of these measures can often signal the need for the. 
staff to. give that student special career and personal guidance. 

I^gjUption deSGribed so far in this section deals with the use of 
individual student pretest data. A group summary of pretest and 
background data can also serve the useful purpose of describing 
the characteristics of tfie students that applied for and entered ^ 
IBCE. If stoilar background data are available from a random 
cross sample of the total high school population from which the 
EBeE students came^ a comparative description can be made to 
determine the extent to which the EBCE studenes are the same as 
or different from those not entering EBCE, This comparison can be^ 
important in determining whether EBCE is drawing students with a 
total r^ngm of student abilities or whether it may be appealing 
primarily to a speqial se^ent of -the student population such as 
those of lower acadgroic ability. In addition t^ such comparisons, 
EBCE staff should look closely at the range and standard deviations 
of pretest and baseline data to determine the amount of variation 
of students' abilities and background e^qperiences* If the scores 
vary widely across students it becomes essential for the program 
to be able to deal with widely different student interests and 
abilities. 
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Use student monitoring data 



As a result of a problem noted during the first several years of 
EBCE operation, the danonatration site itaff felt a need to monitor 
student progress in IBCE in a systematic way, at frequent intervals 
throughout the school year, to insure that students were progressing 
adequately in their project rec^irements, A school year action 
zone system^ was initiated that made clear the level of progress 
expected of students at each of the zones throughout the school 
year. Midway tteough each Eone the staff holds a zone progress 
meeting ta discuss the progress of each student and what help may 
be needed by students. At the end of ieach zone a debriefing is 
held with each student and a report given to His or her parents 
dealing with the student's progress In oompletlng competencies, 
projects, journals, explorations, learning levels, and in meeting 
target dates. Tfiis evaluation feedback to parents and students 
Is ^considered an essential way to help^tudents keep on schedule 
In EBCE* At a group level/ these monitoring data are useful to 
allow the project staff throughout the year to determine if they 
may be overemphasizing one aspect of the progrmi, such as the 
exploration levels, while underemphasizing other areas, such as 
the competencies* 



^ ^ — ~ " CROSS^REFERBNCE -. = — ; 

Pages 66^69 of the EBCE Curriculum ^ Instruatlon handbook aontaii 
more speaific inforimtlon on the ^way in whlah the action zones 
are operated. 

I 



7,; 



64 



Stmps to Follow (Xtmn 25} 




Uae program evaluation data 



In addition to the focus on atudant outcomes ^ an effective program 
evaluation should also look at areas of project management and 
operations where , evaluation information may be useful in guiding 
possible changes or mDdifications in the operations of the progrMi, 
The NWREL IBCE valuation unit has developed two cheGklists--one 
dedling with EBCl essential char aatar is tics ard oUier with 
EBCE -processes—to help identify the extent to which an EBCl sit® 
is operating, in a manner consistent with or different from the EBCE 
operational inodel. ^ese checklists provide for a self ^description 
of any deviations that occur and a reason for these deviations. 

Project staff have found these two EBCE checklists uieful as a tool 
in reviewing an EBCl project proposal to insure that they have 
addressed each of the essential elmients of Ij^erierice-Based Career 
Education. ^After the project is impl^nented for several months, 
some staff have found the two instnarients a useful check to 
determine if they have inadvertently omitted or imder emphasised 
certain areas of IBCTI in their haste; to get underway witii students. 
The Essential Qiaracteristios Checklist has caused several project > 
directors to recofhize that they may have been negl eating areas ^ 
such as systematic ..training of conmunity resource people. 

Some of the mmh EBCE pilot sites have held staff -meetings to review 
the evaluation findings from the prior year. Such reviews have 
sometimes allowed staff to get a more generalized perspective of 
changes occurring in students. Often the Jata confirm hunches held 
by project staff based on their daily interaction with the program. 
At other times , anonymous questionnaire results may uncover either 
positive or negative feelings that students^ parents or employers 
may not have es^ressed direotiy to the project staff. 

A review of ar^as in which students failed to doiionstrate significant 
growth and of perceived weaknesses identified by student^ parent, 
employer and staff qd'festionnaires can be useful in providing 
direction for progrMi modification. A review of evaluation findings 
at one pilot site led to a staff decision to encourage students- to i 
use a broader variety of resources in completing tjieir projects, 
to locate and use student materials that related to applied 
mathenatics^ md to canplete student competency packages that will 
help guid^. students in mastering the progr^'s competencies. 

A review of program evaluation data can also lead to a decision to 
change the evaluation instruments to be used the following year. 
iKperlence frOTi the 1975-76 school year caused the NWREL IBCl 
evaluation staff to develop and recoTOnend the Student Attitude 
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Questionnaire as a more sensitive growth measure for EBCK sites 
than the Psychosocial Maturity Seala or the Self-Directed Search . 

in simanaryp evaluation data can be used to provide feedMck on a 
istudent'a progress, to identify areas for program modifiaation and 
to improve the evaluation in futwe years. 



1 ' 

CROSS-^RBFERENCE 



The EBCE Essential ChmracterlBtlas ChmckllBt &nd EBCE Proamss 
ChmakUst have been descrlbmd on page 52 of this handbook. A 
copy of thesmftwo instrvmentm is located in the R&producible 
MaterialB Emctlon. of this handbook. 
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APPENDIX A I 

i ■ ■ 

ftCHNICAL INFORMATiqN ON ORDERING EVALUATION INSTRUMENTS 



InfoCTiation Gontainad in ^is appendix will be; useful as a resource 
to peraons wlihing to order evaluation instrmenta previously foimd 
appropriate for vmm in evaluating NWOTL ElCE progrOTis, 

^ • I 

Thm addresses anti aosta llstad below are those avail^le to nwKEL 
as of Deoentoer^ 19t6, Pripes are ai^jeot to change by the polisher 

CTBS, Laval 4, Form S : | 

Publisher I CTB/McGraw-Hill t 
Del Mmte Rssearch Park 
Monterey^ California 94930 
(408) 649-8400 



jyiswer 

^-^eet^g^ 



6901 

6902 
6903 

6904 



Code 
No, 



80913 
10957 

10959 



10923 

75901 

75902 
75903 

75904 



Title 



Cost QuantitVi 



^eciiMn Set 

Conplets battery tsst^books 
(Includes 1 Examiner's' Mmual) 
Partial battery test books 
(EKcludes science mnd social 
Btudiea) ^ \ ; 
fading and Bef^rences Skills 
Test (books onl^ 
Bea^iig Mid Refar^nces Skills 
Digitek Answer Sheets 
Language Mgitek Answer Sheets 
Matiiematics Digitek Answer : 
Sheets I 
Soience and Social Studies 
Digitek Answer Sheets 



S 3,00 
19,95 

18.20 



11. 
35 

35 



9.45 


35 


4.50 


50 


4.50 


50 


4.50 


50 


4.50 


50 



Other OTsts, are availiyble by writing the era/McGraw-Hill Company* 



Psychosocial Maturity Scale 

• J " " ^ I 

Pi^lisheri Dr* Ellen Greenberger 

Program in Social Ecology 

Iftiiversity of California at Irvine 
' Irvine, California 92717 

/ - ^ 

■^his instrianmt is undergoing revision and ^ administration md 
technical manual is being prepared by Dr, Greenberger* toformatlon 
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about the ourrent status and cost of tiiia inatrm!tent,M available 
from the teat author. The sooial adequacy aaale of this inatrmnent 
may be most auited to EBCl outcomes. 

Student Attitude Qijestlonnalre 

Publisheri Offica of Dissemination and Marketing 

Northweat Regional Educational Laborato^ 
710 S,w, Second Avmnm i 
Portland^ Oregon 97204 

(503) 248-^6951 i 

This inatrisnent ia available at a price of $,50 per atudent copy/ 
It uses the ^Scan standard answer sheets (DS1120--C) * These forms 
are available for $15.00 par thouaand frOTi 

Precision Busineaa Forms 
120 West 155to Street 
Gardena^ California 90248 

Self Directed Search 

Publisher I Consulting Psychologists Press 

~' ~^^^?=6©i-l«ge-Aveaue_„____^ 

Palo Alto, California 94306~^ 

r 
i 
I 



Self Diraated Oearch Instrument (eonsum^le) 
Occupations Pindar (not eonsumable) 



Cost Quantity 

$30.00 50 
5.50 25 



, APPENDIX B 

DIRECTIONS FOR ADMINISTERING EVALUATION INSTRUMkNTS 



When administering standardised test instrmnents it is very 
important that test adinlnietratioii procedures are followed 
a^q^licitly. Bie nonaaUve data for suah instrinnentation which was 
gathered urider BtandardiEed atoiniatration procedlirea is not 
appropriate for students coitpleting ttie testing under different 
atodnistratiOT procedures, to eKtreme exam|ftLe might involve 
allowing students ujilimi ted time to take a teat which was intended 
to have a reatricted test time period* ^e resultant sooras v;ould 
normally be higher than appropriate for the testing group and the 
normative data would be meaningless. 

EBCE programs are less depen^nt on normative data ad much more 
dependent m estimating stur^t growth d^iiig totf program. In 
m^y inatanceS/ the growth o£ IBOB students is compared to cne 
growth of some competing program or control group of students, 
ttiless all students are tested imder the mmm testing conditiona 
with the aame direations, these ooi^^riaons are meaningless. 
Likewise, if students are pretested at the beginning of ttie program 
with one set of adminiatration procedurea md then postteated at 
the end of the progr^ year with a different set, one would es^^ct 
differences in the scores due to these changes in adminiatration 
prooedures alone. ThuBt such differences become confused with the 
_real ^ffarenoea due to the actual, effect of the program itself , 

The remainder of this docwsent detaila thi^^pntstrati-ve-^procedures 
for selected iMtrumentation uaed in the IBCl program, Tffliese 
inatrinnents inoludei Comprehensive Test of Basic Skills ^ Student 
Opinion Sunrey, Student Attitude guestionnair Student ^pplioation/ 
Badcgrotmd Questionnaire, NWRIL ^mmtic Differential and Student 
End-of-Year Questionnaire, ^ese. administrative procedurea .should 
be followed eKplieitly for ea^ a&niriistratj.on of these inatrianents , 
even if they are highly redimdant for most of tiie students being 
tested. 

If several instruments are being administered, toe teating achedule 
on the next page is suggested. Note that tl*;u most highly standardised 
and tlined instruments are atoinistered firat, Mditipnal instruments 
can then be coinpleted by students on an individual basis # if 
necessary, ^e indicatiid times are based on our eKperiences over 
the past few years and inclu^ ttae for explaining directions. If 
^e entire CTBS is being adminiatered rather tiian only the three 
s^tests we have selected ^ the testing period will have to cover 
several; days* 
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Suggested Testing Schedule 

Required Time (minutes) 

1. eras Beaaing Ca^reheneion 40 
(35 minute timed test) ' ■ '■■ 

2 , Stretch breiJc 2 

3* eras Mathamaties Cpneepts and Applieatione 35 
(30 minute timed test) ' ■ 

4* Conversationar break ' 5 

5. Student Attitude Questiorin^ira 
Seetion i (20 ndnute tijned test) 
Seatdone II-IV 

6, Stretch bre^ 
Heliucation or Imicheon. break 

8. Student ^plicatiDn/Background Qusstinnnaire 
-or- Student Ind-Df-Yaar Quegtiomiaire 

9. Student Cpinion Survey 
10* NI^L IBCE Semantia Differential 

— — _____^tal Pretest Time (3 hrs. 15 min j i95 minutes 
Total Posttest Tim (3 hrs JsrmnrrTXr^TOnu^ 




Note tliat it takes 2% hours to eomplete steps 1-6, ^th 1^ houre ' 
\oT less to be completed after this point* ^us, step 7 is an ideal 
plaoe to disemtinue testing for liuioh with the remainder of the 
testing to be CMiplated in the afternoon after lundh~ In this way^ 
all of tte instrmientation requiring timed subtests and/or special^ 
ansror sheets is completed before ^e major bre^ period, Thm 
remaining instrinnentation requires less rigid atoinistratiai 
Procedures, 
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Comprehensive Test of Basic Skins (CTBS) 

Hie CTBS is a highly standardised instrument with timed oi^tests, 
adsdnietration procedures for tlds instrmnant are thoroughly 
degoribsd and must be .followed aKplicitly* Whan administering this 
instriOTent tte tester(a) mu&Wiave a copy of thm correct CTBS 
^aminar's Stomal, For Level 4, Form S of the CraS this manual is 
dociflnent . v z 

Testers should preview tiie testing process by reading the section 
on "Description of the Tests" on pages 2-5 of the Examiner's Manual* 
This section provides the reader with an overview of the total test. 

Each tester should become thoroughly familiar witA "Part 2, 
Preparation for Actainisteilng CTBS/S" on pages 7 to 12 > These 
general instructions should be followed carefully, with the following 
changes, if only the Reading Comprehension and Mathematics Concepts 
and Applications subtests ^e being administered, as recommended by 
the NTOEL evaluation staff, tiiese si^tests may be atoinistered 
during one testing session. These subtests were selected to minimize 
testing t±tm while maKiini^ing the potential for obtaining student 
growth data, /toius, it is suggested that the Reading Qpmprehension.^ 
subtest be administered firsts followed by a two-*minute "stretchi'^'^ 
break and including mth /the atoinisteation of the Mathematics 
Con ceptTr and Appli cations subtests. If ^^e entire CTBg-ts beings 
adtainis teredo the testing sohedule suggested on page^^ would be mdre 
appropriate i. All of the other '^Special Considerfitions*'^ listed on . 
page 7 should be followed whenever possible regardless of wMch 
subtests are being administered* • 

On the right hand side of page 7 there is a list of eight additional 
considerations which should be followed during testing. Point #7 
is extremely important as yo^ role rust be confined to tiiat of a 
test aAfdnistrator and not include tiiat of a teacher. This point 
=j3LJ,mportMt even after toe testing is completed since any test- 
related coacMng may^ affect student per^^ on future tests. 
The "Testing Cau^ons" listed on pages 8 to 11 further detail 
potential pitfalls during tiie testing procedure, ^e general plan 
for a^inisteting any siAtest of the CTBS is' given on page 12* 

The ^swer sheets suggested for use with the CTBS are the Digitek 
answer sheets #6901-6904 as described m page 10, tfhe Language 
mswer sheet (#6902), imlike th^ other toswer sheets, has two ^ 
^prin ted sides. If any of these subtests (Spelling, Language 
Mechanics or Language Egression) are being administered, side 2 
of this answer, sheet must be used for levels 3 and 4. Also, the 
answer blanks for it&a #41 on toe Science subtest and item #40 on 
the Social Studies subtest are not used since Level 4 has *only 40 
and 39 it^s, ^respectively, on these si^tests, * 

*AlT pmffe refmrmncis in this CTBS description arm to tJim EKmdnmr''s 
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Special Directions for Dlgitek Answer Sheets 

If my of the Digitek answer sheets i#6901 -6904) .are being used, 
the directions below mus.t be followed, if the student Attitude 
Quaitionnaire is not being aAninistered the additional directions 
must also be followed, (If this is the first CTBS answer sheet 
bmLng used, give the fallowing directions. Otherwise^ skip to 
the instructions labelled "[**],'^) 

Ji^t At mil times during this test you are to use a #2 iaad penail 
Now, turn .your mnmr&r sheet sideways so that the top ^of the 
answer sheet is at the right side. [DEWNSTRAIE] Print 
(locf - tion HMie) when it says "school" , (program name) whmre 
It says "teacher" ^[PiVJSE] and (today's date) where it says 
"t^st date." [PAUSE] ' ' 

SAY: Now^ print your last name in the hoKes above the letter 

^ grids, followed by your first name and middle Initlml^ Make 
sure your first nama and middle initial begin in th& 
appropriate spaaes and that your first ' nanm does not exceed 
aijf letters. For example, if your first name is Elizabeth, - 
you would print ELIZAB followed by your middle initial. Are 
there any questions? [PAUSE] 

SAY I Now, blMCken the space to tiie right' of the 4 in the box 

labelled "LEVEL" [PAUSE] and 'the space to the right of the S 
in the box iaheliad "F0Ri4. " 

[If the Student Attitude Questionnaire is not being admnistered 
also use the following special directions. Otherwise i begin with 
the actual test directions.] 

SAY: Now, blacken the appropriate letter box below each letter of 
your^name . Use an oval motion of i^e pencil making sure the 
mark does not tough an adjplhing box. [Demonstrate by 
illuatrating the correct method which should result iri the 
following mark. L H^^li ] Emphasize toat the mark should 
be large enough to almost fill the space from left to right 
but not so large as to touch an adjoininy latter space, 
PAUSE J 

Ne^t, blacken the appropriate nu/nber space indicating your 
grade level in the section labelled "Grade." Use the same 
oval motion of toe pencil. [PAUSE] Now, print the last two 
digits of the year of your birth in the box below the section 
libelled "Ymar" imder "Mirthdate ." If you were born in 
1960, you ^ould enter '^60.'^ Then, blacken the Mpppopria^ ^ 
ir^^fch and year spaces, again using an oval motion of the 
pencil', [PAUSE] Finally, prini your sex as "M" fdt malm or 
"F" for fmnale and blacken the appropriAte Jhtte:T space under 
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the section labmlled "Sex." Are there any quQstions? [Begin 
with the actual test directions.] 

[**] 

SAYi Turn your mswer sheet sideways and print (loaation name) 

mfter "schocl," (progr^i name) after "teacher" and (tray's 
date ) after "test data" as you did on the previous Mswer 
sheet (s)^ Be sure you are using a #2 lead pencil. [PAUSE] 
Now^ print your namm/ last name first, as you did before in 
the appropriate boxes. You do not have to complete any 
other inforimtion. Any questions? [pXuse, Begin with Uie 
actual test directions. ] 

[***NOTEi mmn filling in answer blanks the oval motiou is 
not necassary as the boxes are sufficiently separated on the 
answer sheet* ] j 

Thm general directions for administering the CTBS begin on page 13,* 
Each tester should be totally familiar with these directions. 
men atoinistering the test, read aloud each of the coOTnents 
prefaced by '^SAY^' following the directions in the colored boxes 
'on pages 12 13. Thmn, before beginning each subtest, follow 

the directions in the colored boKes and also read aloud those 
sections marked "SAY-/» The directions for the various subtests 
are given on the pages indicated below. 



Test 


1- 


-^Beading Vocabula^ 


Pages 


15^ 


-16 


Test 


2- 


^Reading Comprehension 


Pages 


17^ 


as 


Test 


3- 


-Spelling 


Pages 


19^ 


-20 


Test 


4- 


-Lmguage Mechanics 


Pages 


21- 


^22 


Test 


5- 


-L^guage Es^ression 


Pages 


23- 


"24 


Test 


6- 


-tothematics Computation 


Page 


25 




Test 


7- 


^Mat±ematics Concepts and 












Applications 


Pages 


26- 


'27 


Test 


8- 


"Beferehce Skills 


Page 


28 




Test 


9" 


-Sclenee^ 


Pages 


29- 


30 


Test 


10' 


-"Social Studies 


Pages 


31- 


32 



The directions should be read verbatim from the indicated pages 
^ove for each si^test during the test administration- Make sure 
the sttMents are using the correct Mswer sheet for each subtest . 
In addition, if only Test .2 (Paading Comprehension) and Test 7 . 
(tothematios Concepts Md ^plications) are being aiministerad, 
some additional instructions are necessary, Oti page 17 at the \md 
of the second "BMt'' section, where it says, "Do' not turn the page 
yet.", add the following s -'^^ . 

M^e sure goii arm marking your answers in the seation of the 
answer sheet labelled' '^Tmst 2, Reading Comprehmnsion," 
There Bhould not bm any marks made in the sections for Test 1 
or Test 8 

*Page referenaes in this seation are to the CTBS Examiner's Manual 
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Ck\ page 26 at the end of the smcond "SAYi" section where it sa^s, 
''jto not go cm to the next page yet^" add the following; 

Make surm you are marking your answers in the s&ctidn of the 
Mnsv\?r sheet labBlled "Test 7 , Mathematics Concepts and 
Applications V' There should not be any marks made in the 
smation for Test 6, 

Following tile test atoLnistration check the answer sheets to 
ensure the presence of student names and make sure all answers 
appear in the correct sections of the answer sheets* - 

Student Attitude Questionnaire 

The Student Attitude Questionnaire is a composite of four separate 
instrwients. The first part is a timed test while students can 
con^lete the otiier three sections at their ovm rate. However, 
all responses go onto one answer sheet* 

' \ '^dtninletration Procedure . Distribu^i^:^^ the ^Scan standard blue 
answer Liheets (M1120-C) to each student. Then follow the 
iristructipns given below* 

ShYis^At all tiims during this test you are to use a #2 ledd pencil 
\Now^ turn your answer sheet sidewaus so that the top Qf the 
hnswer sheet is at the right side* [DITONSTRATE] PzAnt 
^ (location nMe). where It says " school ^" [PAUSE] \ 

(program nam e) where it says "i -uator/'' [PAUSE] tod 
(today's date) where it says "t [PAUSl] 

ShYi Now, print' your last name in the boxes above tfta ietter 

grids, followed by your first name and middlm initial. Make 
sure your first naim and middle initial begin in the 
appropriate spaces and that your first nanm 3es not: exceed 
six letters. For example, if your first name is Elizabeth, 
you would print ELIZAB followed by your middle initial* Are 
there any questions? [PAUSE] 

SAYi Now, blacken the apprBprimte letter box below eaah letter of 
your namm» Use an oval motion of the pencil making sure the 
mark dc&s not touQh an adjoining box, [Demonstrate by 
illustrating tiie correct method which ^should result in the 
following mark. [ ] Ei^hasize that the mark should 

be large enough to almost fill the space from left to right 
but not so large "^-^io touch an adjoining letter space, 
PAUSl] 

SAY I Nc^ J find, the box limlled "Senmster" and blacken the Bpace 
to tha right of "Fall** (if pretest, or "Spring'* if posttest) . 
[PAUSE] 
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SAY 2 mxt, blacken tfte appropriate nm^Qr space indiamting your 

grade levml in the sea ti on l^elled "Grade." Use the sam oval 
motion of the pencil. [PAUSE] Now, print:.^thB Id two 
digits of thm year of your birth in the box below the 
SBctian labelled "Year" under "BirtMate." If you were born 
in 1960, you would enter "60." Then, blacken the appropriate 
month and year spaces, again using an oval motion of tlie 
pencil. [PADSE] Finally, print your sex as ''B" for male or 
"G" for female and blacken the appropriate letter space mder 
the section labelled "Sex." Are there any guestions? [^gin 
with th& actual test directions. ] 

[***NOTEi men filling in answer blmks the oval motLon is 
not necessary as tlie boxes are sufficiently separated on tiie 
Mswer sheet* ] 

The general directions for administering Part I, the New ftexico 
Car^/r Oriented Activities Qiecklist , appear on page 1-1 of the 
Student Attitude Questionnaire > Tell the students to read these 
c^rections to themselves while you read them aloud. Enphasiie that 
response position #5 on the answer sheet will not be used for this 
section -and should not be marked, ^e test is timed so keep an 
accurate record, ^en all directions have been read, tell the 
students to begin but remin i them to stop if they get to the" end of 
Part I. 



Inform the students that they are free to ask the tester questions 
they may have ^ out items on this instrument. This is accomplished 
with the least disruption by having fetudants raise their hand if ' 
ttey^have a question and tJien working with that student on an / 
individual basisV 

After 20 minutes have elapsed, tell the students to "STOP," Tell 
the students to turn to the directions for Part II, the School; 
Opinion Scale, on page II-l. Then, instruct the sludents to read 
these directiona to themselves while you read th^ aloud. Note 
that there are different instruction sets in t^e middle, of the page 
for different test periods. Emphasize that students are to begin 
marking their responses in section II with item #41, 

Before telling students to begin' this section , taxi them uhat they 
are to contiiiue until they reach the end of the test, temind 
them to read the instructions for each part and that the answers 
for each part are placed in a different section on the answer sheet. 
Again, the students are free to ask questions by raising their hand. 
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Student Opinion Scale 

The Student Opinion Scale, also known am the Psychosocial Maturity 
Scale or PMS, is available in two foms. The regular long fom ' 
consists of 102 items comprising 9 scale scores, which convert into 
three a mroiary^ scores , plus a social ^sirability ("lie") scal^i. . 
The second version. Form SA, consists of 42 items comprising just 
3 scale scores, which convert into one suimnary score, plus the 
social desirability (''lie") scale. The atoinistration procedures 
and mswer Lneet are the same for both versions. 

Administration Procedure . Distribute the OpScan standard blue 
answer sheets 'N^S11:^0-C) to each student. If neither the CTBS with 
Digitek answer sheets nor the Student Attitude Questionnaire are 
being administered, use the special directions below. Othenrfise, 
skip to the section marked "[**]. " 

SAY I At all tirms during ^is t&st you are to use a #2 lead pencil 
Now, turn your answer sheet sideways so that the top of the 
answer sheet is at the right side. [DEMONSTRATE] Print 
(location name) where it says "school,*' [PAUSE] Iprogran 
n^e) where it says "instructor" [PAUSE] and (today 's 
date) where it s^ys "test." [PAUSE] 

Now, ptint your last name in the boxes above the letter 
grid^, followed by your Ji£St rmme and middle initial. Mak e^ 
sure your first nake aM middle initial begin in the 
appropriate spaces and thai your first naine does not exceed 
' six letters. For example, if your first namm is Elizabeth, 
you would print ELIZAB followed by your middle initial. Are 
there any questions? [PAUSE] 

Now, find the box labelled "Semester" and blacken the box to 
the right of "F^ll" (if pretest, or "Spring" if posttest) . 

[If both the Student Attitude Questionnaire the CTBS are not 
being administered, . also use the following BpeaiaL^^irmcJ-lmmT^^ 
Otherwise, begin with the actual teat directions.] 

SAY I Now, blaaken the apjprapriate letter-box below each letter of 
your name. Use an oval motion of the pencil making sure the 
■ nmrk does not touch an adjoining box. [tomonstrate by 

illustrating the correct method which should result in the ' 
following mark. [ ][^]i ] Imphasize tiiat the mark should 
be large enough to almost fill the space from left to right 
but not so large as to touch an adjoining letter spaca* i 
PAUSE] 

\ 
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SAY I Next, blacken the appropriate number space Indlcmting your 
grade ievel in the section Ubmlled "Grade." Use the same 
oval motion of the pencil. [PAUSEj Now, print the last two 
digits of th& year of your birth in the box below the 
section laballed Ymar" undmr "Birthdate," If you were born 
in 1960, you would enter "00." Then, blacken the appropriate 
month and year spaces , ^galn using an oval motion ox^ the pencil . 
[PAUSE] Finally, print your sex as "B" for maio or "G" for 
female and blacken the appropriate letter space wider fcha 
section labelled "Sex." Are there any questions? [Begin 
with the actual test directions,] 

[**] 

SAY: Turn your answer ^heet midewayB and print ( iQcation iiam s) 

after "school," (program name) after "instructor". an5 Uoday's 
data) after "test" as you did on the previom answer sheet (e) . 
Be sure you are using a ^2 lead pencil. [PAUSE] Now, print 
your name, last name first, as you did before in the 
appropriate boxes. Also, blacken the box to the\right of 
"Fall" (if pretest, or "Spring" if posttest) • You do not 
have to complete any other infortmtion. Any questions? 
[PAUSE. Begin with the actual test directions,] 

[***NOTOi men filling in mswer blaiJcs the oval motion is 
not necessa^ as the boxes are sufficiently separated on the 
«iswer shecit* ] 

Now a^strlbute the Studen^ pinion Scale test booklets. The 
general dArections for administaring the Student Opinion Scale are 
given on the cover page of this instnment. Tell the studanta to 
read the <^rections to themselves while you read tnem aloua. 
En^hasi^e that response position #5 on the ariswer sheet will not 
Be used and should not be marked. StadBnts than work on the 
instrument at their own pace. 



EBCI Student Appllcation/BackgrGund Questionnaire 

At some sites the EBCE vS;tudent Applieatian/Backgroimd Questionnaire 
will b.^ ::/amlnxstered to ^oapective students before the program 
begins. ^ An otiier sites, the application Eorm will be completed 
by students prior to program entry, hut thm backgre md juestionnaire 
will ke coi^leted during pretesting. 

Regardless of which way the instrument is adm_ i stored at yo^ site* 
the adhniniatr<^tion procedures , are the same, uie student is □in^ly 
told to oott^lete the Student %pli cation fom and secure him parmA^' 
signatures of approval. For the Student Badcgroimd Que^stlonnairt. 
the student is told to r^ad the gv^aral directions at thci l:c^ of 
the page arid complete the mswers to all questions, if back- 
groimd questionnaire is being atoinistered with the a^jpiicatlOTi 
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form already detachad, be s^e to have the students write their 
names in the upper right hand corner before beginning to conplete 
the instrument. Also, tell the s.tudents to ignore the nmidDers 
in parentheses in the left hand margin as these number are for 
data processing pu^oses only, \ 

InfoOT the students that they are free to ask the tester q^ia^iions 
they may have ^out items on this instrment. ^is is accomplished 
.with the least disruption by having students r. ise their hand if 
they have a question and then working with that student on an 
indiyidual basis. 



NWREL EBCE Semantic Differential 

The Semantic DiffeMntial is designed to tap student feelings 
^out certain conGepts,. It is important ttiSt they imdsrstand the 
pu^ose of this instrmient. Before beginning the instriment^ ' 
have the students write their name on the line provided on the 
cover sheet. Then, tell the students to read the directions on the 
cover sheet to themselves while you read them aloud. Students can 
then work on the instrmnent at their ovm pacr , 

Inform tile students that they are free to ask the tester any 
questions they may hrvve abeut this instrmnent, ThLm is 
accomplished with the least disruption by havi n students raise 
tbeii hmd if uhey have a question and then work \g ^^ith that 
student on An in^.vidual basis* ' 

Student End-of-Year Questionnaire 

This instrument will be administered only durii c :ae posttest 
(usually at th^ end of the program year) , Tell zm students to 
write their nroncs^ location (or school district) and testing date 
in the spaces provided at thf top of the questionnaire. Then, 
read the directions aloud while the students read them silently to 
theBiselves* Emphasiaa that the n^rtbers in parentheses in the l^ft 
ha:jid margin are to be ignored as these are used for data 
processing pu^oses only. 

Inform the students that they 3D:e free to ask the tester any 
questions they may ha^'>3 about itmm on this instrisnent* This is 
accomplished with tiia least dirruption by having students raise 
their hand if 'they have a question and th^i working witii that 
atudent on an individual basis. 



PLEASE NOTE: Additional information an .^coring proaedurm far 
the evaluation in^trumBntB may be obtained from the NWREL 
Career Eduaution office. 



MATERIALS 
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I 

/ 



Reproducible Materia Is 



NWREL-DEVELOPED EBCE EVALUATION INSTRUMENTS 



The reproducible materials found in this section are EBCE program 
evaluation instuinents developed by NWHEL. Some of these instruments/ 
such as the Parent Opinion Survey and Snp lover Opinion Survey^ 
contain items that were jointly developed by NWREL and the three 
other educational laboratories involved in the development of EBCE— 
Appalachia Educational Laboratory, Inc., Far West Laboratory for 
Educational Research and Development, and Research for Better 
Schools. School districts uFing NWREL*s model of EBCE have 
permission to reproduce and use these evaluation instruments at no 
charge. 

Listed below are the nine EBCE evaluation instruments contained in 
this section I 

• Empioyer Opinion Survey 

• Essential Characteristics Checklist 

• Parent Opinion Survey 

• Process Ci' icklist 

^ • SOTiantic Differential 

m Staff Questionnaire 

• Student Application/Background Questionnaire 

• Student End-of«Ycar Questionnaire 

m Student Projects Evaluation Forms 

■ ■ i 

The forms, are arranged alpha) atically. Each is preceded by a 
smnmary of the purpose of the form, suggestud data collection ' 
procedures and cover letters (w= ire appropriate) . ThB Student 
Attitude Quastionnaire is nc% mtad here since it coiita^'rs 
copyright sections. T^is instr™ant can ^ purcnased as descrjied 
on page 70. 

The forms -diamselves are printed full-size and "camera-read/'* so 
/ou can u^c own reproducing equipmant to make multtpli sopies. 

They are priu here ■ nt side only to facilital^a reproduction, but 
we reooim^nd that yt u p^int th.i individual forms front 5nd b^ck £nr 
eoonoiiiy, ^ese camBra-t jady forms have been hole«pi;ffiched tor 
inclusiun in ti\is haii^ook; bd sura to mask the holcis before 
reproducing toem. You can mtsk over tha holes with typist's white 
correction t^e or any flat-finiah tap^, which is thep- "whitr d out" 
with stMdard typing correction flu^d. 

i ' ' . . 
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EMPLnYER OPINION SURVEY 



Purpose 

Since participating employers are an essential aspect of EBCE, their 
opinions about the program are important. This questionnaire is 
designed to determine how employers become involve'd with EBCE, how 
they judge the progress of students with whom they have worked, the 
impact of EBCE within their organization and their perceptions 
regarding the operations of.tha program and their role in it. 

Directions for Data Conection 

It is recommended that this anployer Opinion Survey instrument be 
mailed out liver ed by a staff member to each employer instructor 

who has worked with one or more students during the year on either 
an exploration/ learning or skill building level. The Instrument is 
bmst distributed ^n late March or early April. By this time the 
employer instructdr has been able to form an opinion about the EBCE 
program. Approximately two weeks after delivering the surveys a 
check should be made by a y'roject staff member on which employers 
have not yet completed the survey. A fcllowup call or delivery of 
a second copy of the survey should be made when appropriate to obtain 
a larger rate of completed surveys returned. The complete set should 
then be given to the evaluator for analysis. 



U2 
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^SAMPLE COVER LErTER FOR E3CE EMPLOyER OPINION SURVEY 



Since our Ixperience-Based Career Education (EBCE) Progrm^ called 

[name of project] began we have worked with 

approxiinately [nignber of] students. 

Our staff and the evaluation team are interested in finding out 
your opinion about how effective the program has been in providing 
meaningful education to students. Your answers on the enclosed 
Employer Opinion Survey will help us to identify the program's 
successes and failures arid learn where improvements are needed. 

The responses you make on this suivey will be held in complete 
confidence by the eva ligation tean and only a sironary of responses 
from all employer instruntors will be given to the project staff. 

Please complete the enclosed aipToyer Opinion Survey and return it 
to us in the self -addressed envelope by [date] , We will 

collect and forward these xmopened questionnaires to the 
evaluator for analysis. The first one-half page of this survey 
has already been fillad- in to save you time. If there are errors 
m this section/ please make the necessary corrections. If you 
have any questions as you complete the survey, please feel free to 
call me at ftelephone number] . . m 

We appreciate your taking the time to contribute to the evaluation 
effort. Yoiir responses will help us improve the program for current 
students and for future students in such progrOTs. 

Best wishes, \ 



Project Director 



EBCE EMPLOYER OPINION SURVEY 



Site Nigger 

Your Name 

Title — 

Name of company/organigatim 

Type of company/organisation 

Approximate number of Employees at your site 

Length of tima you have been participating with the progrm 

Students in the program Msit mmployer sites for m amreer exploration 
(usually severe days to get an ovBrvlew of a particular job) or for a 
learning level (usually several wmBks or more to work on a project or 
get a more in--depth view of an oaaupation) , 



When the student is on a ca -^er es^loration or learning le^l at your site, 
approximately how many hour week do you typically spend with the student? 

Number of hours per week for oaraer explo-i ition ' 

Nia^er of hours per week for learning level 



Miich of the following supportive services do you (or others at your site) 
providt for the studenta? (CheC}: each mppropriate aategory for career 
exploration and for learnln^j^ level if you have had students for both levels. } 

Career Learning 
EKploration Level 

Talk about job c)pportunities? (15) (24) 

Talk about the student's personal (16) (25) 

problems? ~ " 

Talk about act i\a ties at your (17) (26) 

site? ~ ~ 



Tutor in an academic area? (18) (2 7) 

Evaluate individual student's (19) - (28) 

assignments? ~ ^ 

Assist students in nori'-job» (20) (29) 

related assignments? . ~ \ ~\ 



Supervise students to perfom a (21). , (30) 

specific job-^related *task at your 
site? 



*Plmase ignorm the mmi^BrM in parenthmsms ( ) in the left margin^ Th&ffe 
na^mrs are for data proaessJng purposes only. 



i 



Career Learmng 
Explormtion LevBl 
Help plan sti^ent assignments? (22) (^i) 
Other (plmmBB write in) (23) (32) 



(33) 



3. How did you first become involved with the piogram? (Cheak moprapriate 

^' n Program personnel contacted me about the program 
A student talked to me about the program 
Another employer talked to me ^out the program 
Comp^.y personnel talked to me about the program 
Other (please writm in) 



5.n 



(34) 4. Did the progr^ staff prpvide you with enough in&rmation to help you 

airect student activities at your site? \ ^ 



Yes 
No 



Why? 



(35) 



5 . Would you reconmend to a potential employar or resource person that he/ 
she also baeome invalved with the program? -'\ ' 



2.a 



Yes 
No 



Why? 



(36» 



In general, do you think the IBCE students you have Worked with have 
benefited by being at your site? Cirale the appropriate number from 1 
(definitely no) to 5 (definitely yes) . 



Defini tajy 
NO 



Defini c^iy 
ITS 



(37) 



7, In general * do you think the students you have worked with are really 
interested in learning new things? 



Defini tely 
NO 



Definitely 
YES 



(38) 8, Do you ^k,^ adequate feedback about what lappens to the stuJents 

-^fter th. > ^ve your site? Circle a nmnber from 1 (never) to 5 (MlWi 



1 



(39) 



9- Do you receive adequate fee^ack about the ef fectivenesa of your work 
with the students? 



1 



Alwmys 



(40) 



(4.1) 
(42) 
(43) 
(44) 
(45} 
(46) 
(47) 



(48) 



10. How have employees at your site reacted to your agency *s participation 
in EBCE? (Ch^ak one.) " " 



3. 



□ 



Positive rcdction 
Negative reaction 
Mixed reaction 



6- □ 



^^p reaction 
Not applicable 
Don ' t know 



11, 



In what ways (if any) have the employees at your site benefited? aimak 
onm or more approprimtB rmBponBe (s) t 

1* n They haven't benefited -. 
^ n Increased their awareness of youth - 
3^ O Motivated the regular employees to further training 
Haduced their workload 

'J 

Increased interest in their own work 

I don • t know - . 

Other (please write in) 



4.D 
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Do you plitfi to continue participating 
(ChBCk yes or no^) 

1. □ Y^B No 



in 



the EBCE program next year? 







□ 


(Check one or mort: . thQ reasons 


(49) ^ 


1. 


Progrdtm is worthwhile 


(50) ' 


2. 


□ 


I like thf? people involved 


(51) 


3, 


□ 


My participation i^ a coAmunity service 


(52) 


1, 4. 


□ 


It is challenging to me 


(53) 




□ 


I have had problems with the staff 


(54) 


6. 


□ 


^ I have had problems with the studRnts 


(55) 


7, 


□ 


The program is not effective 


(56) 


8, 


□ 


I don't have time 


(57) 


9, 


□ 


Ochfer {please .^rite in)' 



ERIC 



/ 



/ 



13. What do you think are the greatest strengths of the prograjn? (Chsck 
one or more^^rB&Jons , ) 

(58) tood alternative to a regular high school prograin 

(59) Quality of the staff 

(60) 3,^] Students learn about a variety of careers 

(61) Students learn about real life situations 

(62) S.Q Good way of getting students to learn 

(63) 6.n Sxperience in working with adults 

(64) Other fpiedse write in) _ 



14. What do you think are the greatest weaknesses of the program? (Check 
one or inore rem sons . ) 

(65) \. l-.n Soto students can't handle the freedom 

(66) Problems in the organization of the program 

(67) Students not receiving sufficient training 

(68) Inadequate supervision of students on job sites 

(69) 5.^] Too much paperwork 

(70) 6 . [~1 Lack of feedback about students 

(71) May be too difficult/technical for some students 

(72) 8.0 Other fpiaase htI >e in) 



(73) 15. How many students would your site be able to handle at one time fcr a 

career exploration? 

1 '[ I One student 4. Six to eight students 

2. n Two students 5. Q ^ More than eight students 
3^ Three to five students 

(74) 16 . Wliat do you feel students are abls to learn on job sites that they could 
"^^^ ^ ^ot learn as well in a regular school classroom? (Check one or more 

reasons.) 

(74) l-O First hand knowledge of (77) 4.Q Self-discipline 

demands in a "real world" ' f=i 

(78) 5 ^[ j Motivation to learn 

(75) 2.LJ Working with other people _ 

t^- ^ ^ Nothing 

(76) 3,LJ On-the-job skills . . - n . . . - ^ , 

(80) 7-LJ o^her (please write in) 



17* What other "conments or recomnendations about the program would you 
like to make? 



THANKS FOB YOUR TIME IN COMPLETING THIS SURVEY 



EBCE ESSENTIAL CHARACTERISTICS CHECKLIST 



Purpose 

The purpose of this checklist is to determine to what degree sites 
using the NWREL EBCE program have been able to implement that 
program's essential characteristics. These characteristics consist 
of the basic philosophical and policy features considered essential 
to program organization and operation. Each project director is 
asked to complete the checklist* 

Directions for Data Conection 

It is useful for this checklist to be completed by a project director 
several months after a program has begun and again toward the end 
of the school year. Some project directors have found it helpful 
to have the entire staff discuss their judgment about where the 
program is on certain dimensions of the checklist and to use it as 
a way of monitoring any areas they feel may hive been neglected or 
omitted in the program's operation. The completion of the checklist 
toward the end of the school year can provide an opportunity to 
monitor ways in which the operation of the program may have changed 
over the year. This instrument takes approximately 15 minutes to 
complete* 
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EBCE ESSENTIAL CHARACTER i'.STIC;v CHECKLIST 



(3,4)* Site ' 

(5,6) Respondent _ P^j.,;, 



EKLC 



For eac/i area, rata thB project on a flve^^pcint scnl^^^ using the nc,^o. pointn' 
ind^aatBd ror eaah area. Place the nmntBr 1 to 5 i/Mt tiost ciea ly re^s^ni^ 
your opinion In the box by each area. " " " -^f^^-^-cs 



I. EBCE is an individualized program, 

A. Ongoing staff assessment of student mmds ^ interests and abilities in 
Ba^ic Skills, Life Skills and Careftr L^ivii Icp^nent i 

Q El ^ere is no ongoing assessment in two or morm of these a.:eai. 

Q Student needs, interests and abiiitier are continually a^!ie;ised. 

B. Participation in assessment: 

r^'ED Students play a passive role in tli^ .i^;eBSTO.nt process. 
[T| students play an active role in the assessment proce£.s. 
Individual negotiation : 
^ B student projects ^re preassigned and not subject to negotiation. 

^ Ml student projects allow for negotiation between student and staff. 
D. Integration 

™ Student projects are not integrated with a student's prior 
LJ experiences , 

□ i 

Indiviaual assessment and accomitabiiity are integrated with 
^ program learning strategies when learning plans are negotiated. 

1. AecDuntabiUty standards (a set of learning and behavioral 

Qxpectations for stu^nts as members of the EBCTI * coranunity * ) i 

[Y| few if any accoimtability standards. 

C ^ Clear accountability sta/idards exist arid give students the necessa:ry 
flexibility to meet basic program es^ectations , 

i 

II. EBCE 1s a cortTnunl ty-based prograiTi. 

A* tommum^ nto program planning ar 

(12) □ ^ "^^^^ currently exists fa uaunity input. 

; 0 A systematic mechanism exists for p . mg and utilizing conmunity 

■ input, \ 

*Wease ignow the nimbers in pMrentheses in the left margins. Thase numbers 
for data processing' purposes only. 



(10) 



(11) 



EBCE is a cormuni ty-based program, {continued) 

B, Role of the program advisory board; 

I— I There is no prograin advisory board 

[T] program advisor^ board takes an active role in direction ui the 

program by providing program input, 

C. Cormimity members and student learning ^ 

I— j CGmmunity members are not involved in student learning 
__ activities. 

□ i 

m Commimity members serve as resource instructors and certifiers of 
^ student learning. 



D. Provisior .r cormnunity instructor training/development activitiesi 

□ 



rrn There are no comnunity instructor training/development 
activities. 



m There are at least four, regularly scheduled canmiunity instructor 
training/developn^nt activities each year, 

EBCE i$ an experience-based program and is built from the career activities 
of adults. 

Mode of leamingi 
_ 0 Students are instructed in a passive or school-like mode 

m Active, realistic, lifelike learning activities are provided for all 
students. 

B. Student activity: 



□ 



Students have little or no role in planjiing and scheduling their 
aGtivities. 

fsl Students have the responsibility for budgeting their time and 
managing their daily activities. 



C. Utdlization of resources for learning i 

fTl Secondary resources (textbooks, courses) are given priority . 

H primary resources rDeople, institutions such as libraries and 
museums, events) are given priority, 

_?j^^_^fi^Miy^ ty learnina ' ivltles.i_ _ ---^ - — 

[X] "^^^ ^® nlty i^ ; student 

rgn rtdu^L .-jti.^uiys in the community servt the aary context for 
student learning* 



III. EBCE Is an experience-based program and is built from the career activities 
of adults, (continued) " " " ' '' '•'^"^i^^ 

E. Reference population: 

m Teachers and school work are the prinary referent. 

(20) □ r 

|5| Adults in the world of work are the primary referent* 
Conmiunity learning potential: 

„ No Malysis is made of the learniiig potential of the local 
L=-J communi ty , 

(21) □ : 

There is systematic analysis that envies staff and students to take 
full advantage of the learning potential of the local conmunity. 

EBCE must have its own Identity and i jst be comprehensive and Integrated, 

A. Prograin requirements and processes? 



IV. 



jYj Regular high school requirements and processes are used to determine 

student learning plans, 

(22) ; 

- W<V m ESCT program reguire.nents and processes detemine student learning 

^ tr- LJ plaiis. 

Program completion requirements i 

Program completion requirements are vague unspecified or not 
— dif feren^ated from the regular high school requirements. 

(23) 

[J] Program e^letidn requirements are clearly defined and consistent 
with program goals and local requirements. 

C* Curriculum: 

m The curricula structure includes e^eriences in no more than one 
of the following areasi Basic Skills, Life Skills, Career 
~- * Development* 

(24) LJ i 

m The curriculmn structure includes experiences in all of the above 
L_J areas . 

Survival competencies i 

There are no performance-'based survival competencies in the 
fn program. 

(25) Qt 

[5j There are at least ten performance-based smvival competencies in 
the program* 

E, In terra latedneas of eurriculimi areas and student learning: 

|Y1 Curriculum content areas, such as chemistry^ a^ emphasiiad separately. 



(26) 



EKLC 



^ m anphaais is on interrelated eurrieui™ araas and this is demonstrated 
by the student learning activities* 



i J 

The EBCE program places a major emphasis on the career development of 
students. 

A. Types of commimity learning situations; 

ftere are no employer/corrmuni ty learning sites* 

□ ■ 

(yj Provision is made for different types and levels of learning 
situations at employer/coOTnimity sites, 

Enphasis at learning sites: 

Students are paid for their contributions on employer/conmunity 
„ ^ sites. * 

□ ; 

[J] Studants are on enployer/conimunity sites for learning about careers, 
- " not to earn money. 

C. Career decision making: 

rpj StudGHts are not encouraged t-^ improve their career decision makir4g 
' orocesses. 

□ ; ; 

1^ Students are required to gather information about themselves and the 
world of work and apply this infonnation in career decision making, 

D. ReTfiections on student experiences; 

rjn ^ There are few or no opportimities for student self ^evaluation. ^""^ 

r|l Studencs are encouraged to reflect on experiences and evaluate their 
own strengtoSf weaJoiesses and progress. 



PARENT OPINr^N SURVEY 



Purpose 

The Parent Opinio- Survey is dasignad to assess parentF* perceptions 

ct EBCE project strengths and waaknesses, benefits of the program to 

their son or daug.iter and the extent of their involvement in the 

program. Because EBCE is a program that actively involves parents a 
well as studentfj, it becomes important to assess parental opinions 
regarding the pre grain. 



Directions fov [lata Conection 

A higher return rate of parent surveys is often obtained if the 
survey form is accompanied by a cover letter from the project 
director to the parents stating the purpose of the survey, 
confidentiality of the data and how the results might be used to 
improve the program. It is usually preferable that the parent 
surveys be mailed to parents rather than relying on students to take 
them home. Enclosing a stamped pre^addressed envelope with the 
survey helps insure a higher return rate. Past experience indicates 
that it takes approKimately 20 minutes to complete this survey. 
Distributing the surveys to parents in early April generally qives 
them enough time since their son or daughter has entered the program 
to form a judgment of EBCE. This tinging also allows for a followup 
mailing in early May to parents who have not returned the survey by 
that time. 



Parent's name (optional) 
School district 



EBCE PARENT OPINION SURVEY 



This survey is meant to giVB you m opportmity to ej^ress your 
opinions about the E^^erience-Based Career Education (EBCE) program 
in which your son or daughter has been participating, 

Ntost of thv-i questions are to be answered on a scale of numbers from 
to @ . The phrases at the top and bottom of each set of qi^sticns 
indicate what the scale reans. A may Man somerhing like 
"Definitely No"; if you feel strongly that the answer to the question 
is no, then you should circle the Q . A 0 may meM "Definitely 
Yes"; if you feel strongly that the answer is yes, then you should 
circle the @ , The nmnbers in between (2 , 3 . 4) indicate an 
opinion sonewhere in between "Definitely No" and "Definitely Yes," 
Some scales have different phrases, but they all work the same way. 

Read the phrase abo^ and below the nraibers so you know what ..ch 
scale TCans, then rear^ ich question and circle the nim^er which is 
clw^est Lu your opinion. There are no right or wrong answers; your 
thoughts and feelings are the important things in this * . t}i 
answers pareits give will help detemine how . , i the program is 
doinq noi .nd iinprova it in the future * - tefflMiber to circle a number 
for each iiiem. 



The name EBCE is used throughout this survey becuase the same fom is 
being filled out by parents of participating students at other 
locations thro^hout the comt^ where this t^e of eKperience-based 
career education program is being develop^^i^. For our projectr 
whenever a question or statement in the surv^iy refer? to EBCT 1" 
means the particular EBCE project in which Vwar son or daughter is 
pal icipatlng. 



EBCE PROGRAM PARENT OPIHK'A SURVEY 



7 Z 

Site Number / ) 

1. How well do you feel the EBCE ProgrMi compares oimrall with thi 
past school, as^eriences of your daughter or son? 

Muah ^ Much 

WorsB Setter 



2. If you had it to do over again, would you want your son or 
daughtar to participate in the EBm Program? 

DBfinitBly Pe finitely 

No Yes 



3. How well do you thi^ your son or danqht^r likes the EBCE 
Program con^ared with past sch ^ jnces? 

Muah ^^^^ 
WorsB Better 



4. Have you raceived feiiough information about your son or 
daughter progress in the EBCE Program? 

Definitely Db finitely 

yes 



5. About how 0"ten have you had t^iy cont: ict with any EBCE 
Program stail members? 

Almost Very 

Frmquently 



*Please ignore the numbers In pmrentheBes in the left) 
These numbers are for data processing purposes only/ 



nargins . 



J 



In cofflparison with regular classes how much opportmity did the 
"BCE PrograiA provife your daughter or son for learning about 
occL^ations? 

Much About the Much 

Lbss Same ^ More 

1 2 3 4 5 

What effect, if any, has the EBCT Program had on helping your soi 
or daughter form career plans? 

Deflni tely No Definl t&ly 

Bad Sffeat Goad 

12 3 4 5 



In comparison with regular classes how much opportmity did the EBffi 
Program provide your daughter or son for general learning (i.e., 
Basic Skills and Life Skills)? 

Much About the 

Less Smmo 

1 2 ^3 4 



Much 
More 



In comparison with past experiences in regular classes how motivated 
is your daughter or son to learn in the EBCT Program? 

Much About the Muah 

Less - Smms More 



Before entering the EBCE Program, how often did your son or daughter 
talk to yo^ about what was going on in regular classes? 

Almost AlmoBt 
Never Daily 



How often does your son or daughter talk to you atout what's going 
on in the EBCE Program? 

Almost Almost 
Never mlly 



(18) 
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How many meetings have you attended daring this -^nool year where 
other parents of EBCE students were oresent? 



None 



4 or mora 



13. 



(19) 




(20) 




(21) 


□ 3. 


(22) 


□ 4. 


(2 3) 




(24) 





What do you think are the greatest weaknesses of the EBCE Program? 
(Check any of the following which are^pplicablB . ) 

Some students can't handla the freedom 

Problems in the organization/staffing of the program 

Students not receiving sufficient training in basic skills 
or survival skills 

Inadequate si^ervision of students on job sites 

Lack of a variety of job sites to meet students* 
interests 

Other (pleBMB writm in) 



What do you think are 'the greatest strengths of the EBCE Program? 



I Check any of the following which are appliaabjQ 

(25) 1. Good alternative to a regular school program 

{26) Quality of the staff 

(27) n 3. Students learn about a variety of careers 

(28) C]4. Students leam about ^" real life'- situations and 

responsibility 

(29) (tood way of getting students to learn 

(30) nie. Experience in working with adults 

(31) . CD 7- Individual treatment of students 

(32) n 8' Other rpiease write in) 



EKLC 



15. What positive changes, if any, have you noticed in your son or 

daughter that you feel are a result of participation in the EBCE 
Program? (Check one or more of the following. ) 

(33) Q 1, Greater maturity or self direction 

(34) D 2. Batter able to relate to others . 

(35) ^ CD 3. Greater self confidence 

(36) -0 4, Clearer direction about his/her future 

(37) Q 5* More interested in education 

(38) L3 6* Better understanding of jobs 

(39) ^ 7* Improvement in basic skills 

(40) [D 8. More realistic attitudes toward life, work, etc. j 
{41) O 9- None | H . , ^ 
(42) 10. Other (piaase write in) 



16. What naqative changes, if any, have you noticed in your son or 

daughter that you faei are a result of participation in the EBCE 
Program? (Check one or more of the following.) 



(43) [ ) 1. Less interested in aducation' 

(44) ^] 2. Lass interested in working 

(45) Q 3. More critical of others 

(46) ^] 4. Disappointed with the prograin 

(47) Q 5. None 

(48) □ 6. Other (plBasB write in) 



17, What types of knowledge, skills or attitudes have your son or 
daughter acq'uired in the ESCT Prograiti that you feel ha or she 
would not have gotten from a regular high school prograin? 
(Check one or more of the following . ) 



(49) 


□ 


1. 


Firi t-hand knowledge of demands in a *'real world 


(50) 


□ 


2 . 


Working with other people 


(51) 


□ 


3. 


On-the-job skills 


(52) 


□ 


4. 


Se If-discipline 


(53) 


□ 


5. 


Motivation to learn 


(54) 


□ 


6. 


Nothing 


(55) 


□ 


7. 


Other ( please write in) 



(56) 18, What changes, if any, would you recoir^nend in the EBCE Program? 



EBCE PROCESS CHECKLIST 



Purpose 

This checklist Is Intanaed as an instruiTient to identify areas in which 
EBCE sites are consi.^Cent with or different from the NWREL EBCE 
operational modal* Deviations from NWREL procedures are viewed here 
from a neutral perspective in that certain deviations may produce 
different results 'than thosa described in the NWREL EBCE handbooks, 
A description of what deviations occur, their reasons and their 
results is essential for an understanding of the EBCE implementation 
effort. This checklist covers four areasi program objectives* 
management and organization, curriculum and instruction and student 
services processes* 

Dlractions for Data CDHection 

It is useful for this checklist to be completed by a project director 
several months after. a program has begun and again toward the end 
of the school year. Some project' directors hav^t found it helpful 
to have the entire staff discuss their judgment about where the 
program is on certain dimensions of the checklist and to use it as 
a way of monitoring any areas they feel may have been neglected or 
omitted in the program *s operation. The completion of the checklist 
toward the end of the school year can provide an opportunity to 
monitor ways in which the operation of the progrmi may have changed 
over the year. This instrument takes approximately .20-30 minutes to 
complete . . ^ 



EBCE PROCESS CHECKLIST 



(3,4)*S1te 
(5,6) Rispondent 



Date 



OBJECTIVES 



For maah obj&ativB listed below pXmaBe check whmther It applies to all, some 
or none of your EBCE students. Also list any additionml student outcomes that 
your project may have, (Please note that the Life Skills curricu}um area is 
not included here because it is addressed in Sectlion III B.) 



(7) 



(8) 



Career Developmerit 



students v/ill increase their knowledge of 
their own aptitudes^ interasts and 
toilities and apply this understanding to 
their poterjtial career interests* 



2 . Studants will increase tiipir knowledge of 
social, governTOntal and economic issues 
and trends in the world of work. 



3* Students \^ill develop the general skills 
of job finding, job application, on-the- 
job negotiation and dependability 
(9) neeessiiry in daily work interactions* 



4* Students will analyze potential careers 
for financial and psychological 
inducements f preparatioii needs and 
(10) preparation progrms available* 



USED WITH^ 
All Some ^ No 

St uden ts St uden ts St udents 



Basic Skills 



. 5. Students- Will improve in their performance 
level of fimdayRiental basic skills (reading, 
writing, oral coiranunication and 
(11) mathematics) . 



Please ignore the nwnbers in parentheses, 
processii^ purposes only. 



These numbers are for data 



1 



U 



6, StudentE will be able to pertom applied 
skill tasks related to qdmmtB of 
(12) interest to them. 



7, Students will bacorne aware of the level of 
basic skills naaded to enter eareers of 
interest to them and will undarstmd the 
ralationihip of that lav&l to their currant 
(13) basic skills proficiancy. 



8. Students will denK^nstrate an increasad 
willingness to apply basic skills tp work 
(14) tasks and to everyday problems* 



ExpiHentlal Outcomes 



9, Students will broaden the range of sourcas 
they use (paople, e^^nts, institutions, 
laws, books, etc*) in gathering infomation 
(15) for work and decision m^ing. 



10, Students will demonstrate thm ability to 
conduct conversation with an adult that 
reveals the student's self-confidenca md 
mderstanding of the other person's message 
(16) and fea lings* 



11* Students will damonstrata m ineraase in 
self ^initiated behaviors and in assmning 
responsibility for earning out and 
evaluating tasks which they agree to 
{17) completa* 

12* Students will demcmstrate an increase in 

behaviors that reveal a tolerance for people 
and institutions having different values, 
ideas or baskground than themsal^s; an 
openness to changa aiid a willingnass to 
(18) trust others when circimistanaas warrant. 
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StudGnts- 



USED WITHt 

Some No 
Students Students 



13, Students will include data from their tots I 
senaory system as part of their input into 
(19) their dacision-m^ing procasees , 



(20) 



14. Students will be ^le to assiro adult 
responsibilities and relationships in a 
positive and self-confident maniier. 



(21) 



15 , Students who select a career area to / 
pursue will acquire specific job skills 
while at employer sitaa related to their 
career afea« 



C22)^6* —Other outcomes fpiease list) 
(23) 

(24) ^^ZZ^^^~Z^^^^~ 

(25) " " " - " " • 
(26) 



II. MANAGEMENT AND ORWNIZATION PROCESSES 



(27) 



Plmmsm chmck thm appropriate reBponmB that deMdribeB the current status of your 
projmat on mmah of th^ iollo^inff dinmnsions* / 

1, Has the district school board approved the project? ' 

n Approved in writing Q 2. Verbally approved Q 3^ Not approved 



2. Has the state department of education approvad the project? 
(28) i _ 

Lj 1. Approved in writing Q 2, Verbally approved Q 3, Not approved 

3. Does the project meet all legal and f^ir labor practice requirements? 



{29) 



n 1. Yes Meets most requirements Q 3, No 



(If you select "meets most requirements" or nOi" please explain.) 



: I 



112 



4* Are the following staff roles being used? (Ch&ak thosa roles actumlly 
defined mnd used* J ' . 

(30) □ Project director (34) Q Student coordinator 

_ (31) □ Learning manager (35) □ Earning aasistanf; 

(32) □ Employer telations specialist (36) Aides 

(33) n Learning resource specialist (37) O Others (list) 



III. CURRICULUM AND INSTRUCTION PROCESSES 



A. Competenclis 



Of thm following aompetmnaimm ch&ak thoBB , if any^ that part of your progrMm. 
List any comp&tenciBS your site may have added, indicmtQ if they , are required of 
all students or are optional. If your site uses cowpetencies differently than as 
described on pages 331-407 of the NWRBL EBCB Curriculum & Instruction handbook, 
pi ease explain the differences on the back of this sheet\ 



1, Transact business on a 

(38) credit basis 

(39) 2* Maintain checking account 

3. Provide adequate insurance 
for yourself, family and 

(40) possessions 

4 i. File state and federal 

(41) taxes 
» 

5* Budget time and money 

(42) effectively 

6* Maintain good physical 
health and m^e effective 

(43) use of leisure time 

7. tespond appropriately to 
fire, police and physical 

(44) health emergencies 



Used 

DiffmrBntly Thmn 
Required Optional in the Handbook Not Used 



(45) 



8. Participate in the 
electoral process 



(46) 
(47) 
(48) 

(49) 
(SO) 



(51) 
(52) 
(53) 



(57) 
(58) 
(59) 



' Used 
Dlffermntly ThAn 
Requizmd Optional in the Handbook Not Used 



9, Understand the basic 

structure md function of 
local government 

10* Explain p**rsonal legal 
rights 

11. Make appropriate use of 
public agencies 

12 * Make applicatibn for 

anployinent and suacessfuyi 
hold a job ' 

13- ^sei^ate and mairitain an 
autoipfcbile 

14 p Othmt cQmpetencie* (ple&s^ 



\ 































~~ — ~ 
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I, Chffak those projuats^if any, that are part of yc^ur program^ List any 

studant projects uofi^ site may have, added ^ indic^tB if they are required of 
all students or are ^optional. If youi site uses Btudent projeats differently 
than described on pig^s 189=373 of the NWREL EBCE Curriculum ^ instruation 
handbook, pl^^psm t^j^plain the differehGos on the back of this shoot . 

^ / . Used 

/ Differently Than 

Reqidred Optional in the Handhook Mot Used 



.OiTOj 



J* Lx.'i'Mivw- dav*%lppiftent 
il^dimduai. project 

3. Criticay Uiinking 
pkredesifned project 

4, Critical thinking 
individual project 

5* S*unatiDnal Gitizenahip 
predasl^ad projeet 

6, Functional citi unship 
indi^diual project 



il4 



(60) 



(61) 



(62) 



(63) 

(64) 
(65) 
(66) 



?• Parsonal/sQcial 
davalopment s 
pradesignad project 

8 , Pa rsonal/sQci al 
development 
individual project 



\ 



9* Science 

pradasipied projact 

10. Scianee 
individual project 

11, Others 



Used 

Differently Than 
Required Optional in the Handbook Not Used 



(67^68)2* How many projects Lb each studant ej^ected to complate each yaar? 



3. On individually prepared Life Skills projects, who generally does the 
following? (Chmak TBsj^nsmB that apply) i 

(69) 1*" Salects the, topics ^ Q l, students Staff ^^^^ 

2i Datanfdnes tha objacti^s and 

(70) activities 

(71) 3. Evaluates the results 



Ql, Students Staff U}^^ Both 

1, Students I |2. Staff Q 3. Both 



C, Exploration Package.^ y 

(72) 1* Are the exploration packages — 

Reqiiired of all students 
Raqiiired of soma students 
Optional for stude^nts 



□ 1- 

□ 2- 

□ 3. 

□ 4. 



Used .differently than dascribed on pages 105-185 of the NWMIL EBCT 
Curriculian Si Instruction handbook. If sq# pleasa eKplain the 
dlfferances. 



{73,74) 



2, How iftitfiy es^lorations , if anyf are required of studMts in your program? 




3. Who selects the ej^loration sites for an individual student? 
{75) 

\ O ^* staff 

Q 2* Ihe student 
r] 3. Staff and student jo:lntly 
^) 4. Otiief (please &peaify) 

(76^77)4* imat is thm average lengyi of each exploration in hours? 

D. Ltarning Livel Process 

(7) 1* Are. learning levels^ — 

1. Required of all students 

' n 2. Required of some students 

n 3. optional for students 

4* Used differently than described on pages 277-327 of the NWREL 
IICE Curr i eul ign & In s tj' uc t i on handbook. If bq, please eKplair 
the differences* 

(8) 2. How many learniufT level ej^eriences * if any* are required of students in 

your programV 

(9 AO) 3* What is the average length of each learning level in hours? 



E. Student Journals ^ 
(IX) 1. What are the primary pturposea served by 'the jouruals? 

\27\ 1p Analysing and integrating career awareness information 
n 2* Developing coromimrrtt ions skills 
1^ 3, Helping students know themselves better 
fr~] 4. Developing trust relationships with an adult 
5, Other (pJease list) ^ . 

(12)2* A^e student jo^nais^- ^ 



Required of all students ' i 
Required of some students 

Optional for students v , 

Used differently than described on page's 411-451 of the Curriculum a 
Instruction handbook. If so* please explain the differences. - 
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^j^r^l2) 3, Approximately how often are students required to write journal entries? 

Q l. Daily Qa. Weekly Qs. Biweekly Monthly Os, Other 

(14) 4. How often are students required to t u^rn in their journals to a staff member? 

ril. Daily Weekly Q 3. Bimonthly Q4, Monthly Other 

F. Studint Seniinars by Employirs ^ 

1* Approximately how many seminars with employer or community resource people 
(15/16) have you held for students or do you plan to hold this school year? ' 



G. Skill Building Level 

(17) 1* . Will skill building levels at employer sites be-- 

Q 1. Required of all students ^ 

2* Required of s«ne students 

3, Optional for students 

iJ 4. Used differently than described on pages 277-327 of the NVTOIL 
EBCE Curriculmn & Instruction handbook. If so, please explain 
the differences p 



(18)2, If skill building levels are m intended part of your program, have my 
; students begun them yet? Q 1, Yes Q 2, No 



H, Special Placements 

(19) 1* Are "speeial placements" of students at enployer or coimmLty sites for 

Life Skills or Basic Skills option within your program? 

. Q 1* Yes []] 2. No * » 

(20) 2* If "speci,al placements" are a part of your progrMi/ have any students 

begun them yet? ^ Q] 1* Yes n 2 , No 



I, Sptqiflc Curriculum Materials 

Please indiaMtm which of the fallwing materials arm used by studmntB In your 
progrmm- 

USED BY: 
All , Same No 
Studmntm Students StudentM 



1, Indi '^dualized beaming for Adults (ILA) 

(21) lasic Skills materials 

(22) 2,' Career Infomation System. (CIS) 



il7 



I' 



(23) 
(24) 
(25) 

. J. 

(26) 1, 

(27) 2. 



3, Other matarials (please speaify) 



USED BY: 
All Somm No 

Students^ tudent& Students 



Progfam Completion Riquiremtnts 

Does your projaat hava written pregfram complation raquiramants that ara 
elaarly d^iinmd? Q l,Yaa 2, No 

If yea, ara the raquiremanta like those described on pages 52-53 of tha 
mmL EBGE Curriculum s InstruotiTO han&ODk? Q 1, Yes Q 2, No 
I£ no, please deseribe the differencas. 



IV, STUDENT SERVICES 



A, 

(28) u 
(29) 



Student Recrultmtnt 



Has student racruitmant bean aimed at a eross saction of local high school 
students? Q 1, Yaa Q 2* No ^ 
If no, what types of students have been re or ui tad? 



(30) 2, What recruitjnent strategies were used? . ^ 

(31) 3* When ware students recruitei? ' 

(32) 4. Wnat types of students actually entered tha program? (IndlamtQ only if 

diff&rent frmi type ^of student recruited^) 



8* Classes 

(33) 1, Are students allowed to take classes at the local high school? 
n l*Yes Q 2. No 
(34,35) 2. If yes, how many students are taking sueh elassas? 
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{36) 2. Are students alla^?ed to take clasaes at comnunity colleges or otlier 
institutions? □l,Yes Qa. No If yes, how many stud^.nta are 
(37p3S) taking such elasses? 

l(39) 3. Are students allowed to take classes or copses at employer sites? 

n 1* Yes Q 2, No If yes, how many students are taking such 
(40,41) classes? 

C* Guidance 

1. Is the guidance function shared by all professional staff mentoers? 

(42) l..Yes Q2,No if no, how is the function handled? 

(43) 2. Do staff mentoers conduct student staffing sessions regularly to discuss 

the progress of each student? Q 1. Yes Q 2, No If no, how do 

staff share information toout students? 



D. Accountibillty Systini 

(44) 1, Does your project utiliEe a student accountability system with clearly 

defined expectations md consequences? Q 1. Yes Qa.Nb 

(45) 2. If yes, does your acOTuntability system work like that described on 

pages 77-91 of the Curriculum s Instructicyi handbook? 
Q 1 , Yes Q 2 . No 

If no, please describe the differehces* * 



E. School Ytar Action Zones 

1. How mmny, if any, school year action zOTes for students are utilised 
(46,47) by your project? 'If nme, write "0*" 

(48) 2* If you have action Eones, are Aey organized like tiiose described on pages 
81-84 of the JMWBEL EBCT jdurricuimi St Instruction hmdbook? 

1. Yes Q 2. No ! 
If no* please describe the differences, ^ ^ 




F. Assessirent Forms 



Lis ted bmlow mrm m number of EBCE forms or inBtruments Bomtlnms used. 
Plm^Bm in^aate which fornm are rmgulrmd, option&l or not usmd* Also 
indicAtm if thmy are revised or different from those developed or ^ad ^ 
by NWREL. 




1. CTBS Reading & Arithinetic 

(49) Subtests (Ci27,54)* 

2. Basie Skills Prescription Pad 

(50) (Ci635) 

3* Belt Diracted ^arch 

(51) (Cai6, 145, 640-41) 

(52) 4* Student Attitude Questionnaire 

(53) 5* Sammtic Diffarantial 

6* Peychosocial Mati^ity Scale 

(54) (Studmt Opinion Scale) 

(55) 1- Parent Opinion Survay 

(56) 8. En^loyer Opinion Survey 

9*, Student ^plication Form 

(57) (Si 69-75) 

(58) 10. Staff Quastionnaire 

(59) 11* End-of-Year Studant Quastionnaira 

12, Learning Site Analysis Form 

(60) (C I 72-75) 

13, Skill DavelopMnt ^cord 

(61) (S 1108-109) 

14* Stutont Performance Baview 

(62) (by employers) (Sai2-113) 













































































/ : 
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j/ 

- / ^ 








/ 















*iliis form is shovm or discussed in tha designated NW^L EBCE handbook on 
these pages* (C^ Curriculim i Ingtruction i S ^ Studant Services) ^oae 
' not referanced will be foimd in the Program Evaluation handbook. " , 
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t63) 

(64) 

(65) 
(66) 
\ (67) 

(68) 

(69) 

(70) 
(71) 
(72) 

(73-75) 



15. Student ivailuation of Learning 
Siti (SillO-111) 



16, EBffl Record of Student Perforinance 
(f :265=297) 

17. EICT Studdnt Is^erience Record 
(Sil02-104) 

18, Weekly Time Reports (Si90-91) 

19. \Student Profile Sheet (Si98-99) 

20* ^AccQuntability Writa-^ Form 
(S 1118-119) 

21,, Learning Site Utilization Form 
; '(Sil06-107) 

22 • \ Maintenance Visit Record 
i (S 1130-131) 

23* j Zone Debriefing Form (SillS-ll?) 

i 

24* I Fredesigned Projects (C:196) 




Plmmsm list miy additionMl £orm or instruments your proj&at uses 
and mttaah m copy of mmah to this ah^akllst. 
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: SEMANTIC, DIFFERENTIAL 

Purpose' . 

The Semantic Differential i s an instriOTent that measures in an 
indirect way students' faelings about certain concepts. In this 
case, the concepts of me, school , adults , l earning , work , aecislon 
making and community resources were chosen by the demonstration 
site stejff and a clinical psychologist as central concepts in the 
EBCE program. Using a five-point scale, students rated each concept 
in terms, of the following 15 polar adjectives* interesting^boring , 
unfriendly- friendly, good-bad, easy-difficult, scary-fun, tense- 
relaxed,! ^s^sonable-unreasonable , sad-happy, wiae-f oolish, 
i4:Xtleva|nt-re levant, open-closed, painful-pleasurable, important- 
unimportant, weak-strong and warm-cold. Significant increases have 
generally been shown on these . concepts between pre and posttest 
sessions for EBCE students, thus reflecting a change in feeling 
about these concepts , 

Directions for Data Collection 

- i ■ . 

The Semantic Differential can be administered to EBCE students as - 
a group .during the pretest session and the posttest session. This 
is not a timed test and students should be reminded to work rapidly 
and record their first impressions rather than taking time to 
reflect jat length on any pair of adjectives. Students should be 
reminded that if they feel a pair of words does not apply or are 
undeciddd, they should place the "k" in the center space rather 
than leave the\line blank. 



119 



Name 



Project or 
School District 



Date 



SEMANTIC DIFFERENTIAL 



On each of the following pages there id a differant topic for you to 
describe* Your description can be made by marking the list of words 
on the page, TiM a look to see how this is dene* Each pair of words 
forms a f^cale. 

For exaniiple^ if you feel that the topic named at the top or the page is 
vegy c 1 OS e ly a s s PC i a t a d with one end of the scalra, you would place an 
"x" as follows: 



(fair jij ' i t ^infair) 



or 



(fair. ; : i i n unfair) 



If you feel that the topic nained at the top of the page is only slightly 
rela ted to one or the other end of the scale ^ you would place an "x" as 
folIwSi 

(fair t X g t % TOfair) [orf (fair i _j i x i unfair) 

If you consider both sides equally associated with thB topic named at 
the top of toe page^ you would plane an "k" in the center spnce as 
follows I 

(fair I K I I unfair) |or| (fair i ■ x i i unfair) 



Rmnmmber: Nmver put more than one "x" an any scule. And mlBO be sure to 
check every item. If you fm&l that a pair of words does not apply ^ or if 
you arm undecided^ place the "x" In the center spacej do not ^^avm the 
line blank. " 



i 



ME 



INTERESTING 



BORING 



UNFRia^DLY 



FRIENDLY 



GOOD 



BAD 



EASY 



DIFFICULT 



SCARY 



FUN 



TENSE 



RELAXED 



REASONABLE 



UNMASONABLE 



SAD 



HAPPY 



WISE 



FOOLISH 



IRRELEH/ANT 



RELEVANT 



Qi^^N 



CLOSED 



PAINFUL 



PI£ASU^BLE 



lOTORTANt 



UNIMPORTANT 



WEAK 



STRONG 



WARM 



COLD 



EKLC 



l2i 



ADULTS 



INTERESTING 



BORING 



UNFRIENDLY 



FRIENDLY 



GOOD 



BAD 



EASY 



DIFFICULT 



SCARY 



FUN 



raiNSE 



RELATED 



REASONABLE 



Ul^ASONABLE 



SAD 



HAPPY 



WISE 



FOOLISH 



IRMLEVANT 



RELEVANT 



OPEN 



CLOSED 



PAlNFUIi 



PLEAS URABI^ 



IMPORTANT 



UNIMPORTANT 



WEAK 



STRONG 



WARM 



COLD 



EKLC 



SCHOOL 



INTERESTING 



BORING 



UtC^RIENDLY 



FRIENDLY 



GOOD 



BAD 



EASY 



DIFFICULT 



SCARY 



FUN 



TENSE 



raiLAXED 



REASONABLE 



UNREASONABLE 



SAO 



HAPPY 



WISE 



FOOLISH 



IRRELEVANT 



RELEVANT 



OPEN 



CLOSED 



PAINFUL 



PLEASURABLE 



IMPORTMIT 



T 



UNIMPORTANT 



WEAK 



STRONG 



WARM 



COLD 



LEARNING 



,TERESTING 



BORING 



UNFRIENDLY 



FRIENDLY 



GOOD 



BAD 



EASY 



DIPFICULT 



SCARY 



FUN 



TENSE 



MIASONABLE 



UNMASONABLE 



SAD 



HAPPY 



WISE 



FOOLISH 



IW^LB^ANT 



MILEVANT 



OPEN 



CLOSED 



PAINFUL 



PmASURABLE 



IMPORTANT 



UNIMPORTANT 



WEAK 



SITONG 



WARM 



COLD 



EKLC 



DECISION MAKING 



INTERESTING 
U!IPRI©IDLY 



BORING 



FRIENDLY 



GOOD 



BAD 



EASY 



DIFFlCUm' 



SCARY 



FUN 



TONSE 



RELAXED 



REASONABLE 



UNREASONABLE 



SAD 



HAPPY 



WISE 



FOOLISH 



IRRELEVANT 



REMIVANT 



0PM 



CLOSED 



PAIOT^L 



PLEASURABLE 



IKPORTMIT 



UNIMPORTANT 



mm 



STBOm 



WARM 



COLD 



EKLC 



12 



^3 



^- COMMUNITY RESOURCES 



ERIC 



INTORESTING 



BORING 



UNFRIENDLY 



FRIENDLY 



GOOD 



BAD 



EASY 



DIFFICULT 



SCARY 



FUN 



RELAXED 



REASONABLE 



UNMiASONABLE 



SAD 



HAPPY 



WISE 



FOOLISH 



IRREL^AOT 



.^LEVANT 



OPEN 



CLOSED 



PAINFUL 



PLEASURABLE 



IMPORTANT 



UNIMPORTANT 



WEAK 



STRONG 



WARM 



COLD 



^^0 



WORK/WORKING 



INTERISTING 



SORING 



UMFRIENDLY 



FBXmDLY 



GOOD 



BAD 



EASY 



DIFFICULT 



SCARY 



FUN 



TONSE 



REASONABLE 



UNREASONABLE 



SAD 



HAPPY 



WISE 



FOOLISH 



IRMIiEVANT 



MLBVANT 



OPEN 



CLOSED 



PAINPUL 



PLEAS UBABLS 



IMPOHTANT 



UNIMPORTANT 



mm 



STTONG 



WARM 



COLD 



STAFF QUESTIONNAIRE 



Purpose 

This questionnaire asks the staff to rate the importance and 
perGaivad effectiveness of leaming strategies used in EBCE. it 
also contains questions dealing with the staff's perceptiron of 
factors contributing to and tlioae limiting the success of the 
program^ changes they would suggest in the program and areas in 
which they feel students have made greatest and least growth. 
Because thm staff are working with thm program on a daily basis , 
their perception of strengtiis and weaknesses ^d their recormnenda^ 
tions for change are important to consider* 

Directions for Data Conectlon 

'Kiis, questionnaire can be distributed by the EBCl project director 
to the staff for return in a sealed envelope to the project 
secretary for a return mailing to the evaluator. Thm envelope 
should state on the outside the staff meniber's name. It takes 
% approximately 15 minutes to complete this form, A week after toe 
questionnaires have been distributed the project director should 
check to see if all staff have completed the questionnaire and 
request that any outstanding questionnaires be returned* 

The suggested memo on the following page may be useful as a cover 
letter* 



SAMPLE MEMO 



MEm TOi 



mCE Project Staff 



FROM: 



Project* Director 



EBCE Staff Questionnaire 



Please take a few minutes to read and coff^lete this staff 
questionnaire. It measi^es your peraeptions of the ijriportance 
and effectiveness of key learning stratagies, the extent to which 
leM'ning goals have been attained by the program and overall 
asseasnsentB of the program* ^e recording of staff evaluations 
of prograHi effects is a very important aspect of tiie evaluation 
of ESCEy and tl \ J :^ . ;_istionnalre is the primary source of these 
data. 

Please complete thLs questionnaire by 

and give it to the project secretary in the sealed envelope for a 
return mailing to the evaluator* 
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EBCE STAFF QUESTIONNAIRE 



a)* 

(3) 



7 4 



Site nufli^er 



Listed below are major learning activities used in one or more mcE programs. 
Please rate e^ch, first in terms of how important you feel it is for' EBCE ' 
Students, and secondly in terms of how effective you feel it has been this 
year- Some activities may be considered important, but not producing 
effective results while others may be considered vey^^'^rfeetive but of low 
in^rtance. If the activity is not used in your prlject* please circle na 
for not applicable. Circle 1 for not important or ^t effective to 5 for 
highly important or highly effective. 



HOW imOKTANT 
Not Highly 
imp* Imp ^ 



( 7) 


41- 


Student orientation 


1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 


( 9) 


b- 


Student Account^iiity System 


1 


2 


3 


4 


5 


1 


2 3 


4 S 


NA 


(11) 


c. 


Student negotiation 


1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 


(13) 


d. 


Predesigned projects 


1 


2 


3 


4 


5 


1 


2 3 


4 S 


NA 


(15) 


a. 


Negotiated projeGts 


1 


2 


3 - 


4 


5 


1 


2 3 


4 5 


NA 


(17). 


f. 


Student journals 


1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 


(19) 




CoBpatancies 
Career ej^lorations 


1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 


(21) 




1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 


(23) 


i. 


Learning level process ' 


1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 


(25) 




Special placements 


1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 


(27) 




ILA Materials 


1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 


(29) 


1, 


^loyer seminars 


1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 


(31) 


m. 


Student retreat 


1 


? 


3 


4 


5 


1 


2 3 


4 5 


NA 


(33) 


n* 


Greap aoti i,tj.es (e,g*^ cadres) 


1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 


(35) 


o. 


Others fpJease specify; 


1 


2 


3 


4 


5 


1 


2 3 


4 5 


NA 



*Plmasm ignore the manbe^j in parmnthmsis in thm left mrgin. These 
numbers ars for data proaessintf purposes onlii. 



ERIC 
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On the grid below please indicate Iww" helpful you feel EBCE mx^rienaes have 
been in helping students to do emch of the following^ (Circle 1 to 5 for eac 
stMtmmBnt) 



(37) 2* Solve problems logically. 1 2 3 4 5 

(38) 3, Understand the role of science 1 2 3 4 5 

in our soeiety today, 

(39) 4* Understand more about 1 2 3 4 5 

theMelves . 

(40) 5* fet along -with others. 1 2 3 4 5 

(41) . 6* Understand the democratic 1 2 3 4 5 

proeesa, 

{42) 7* DeTOlop their own creativity. 1 2 3 4 5 

(43) 8* Learn how their intereata ^d 1 2 3 4 5 

^ilities fit into ^tantial 
careers* 

(44) 9* Liearn hw BQciety*s values, the 1 2 3 4 5 

govemment and the e con any 
affect the world of work. 

(45) 10* Learn to analyse potential jobs. 1 2 3 4 5 

(46) 11. Laarn how to find and keep a 1 2 3 4 5 

job* 

(47) 12, Learn the basic skills 1 2 3 4 5 

necessary for the careers that 
interest them* 

<48) 13* In^jrove their reading skills* 1 2 3 4 5 

(49) 14. Improve their math skills. 1 2 3*4 5 

(50) 15* Inprbve their oral conmiunication 1 2 3 4 5 

, skills* 

(51) / 16. i^ovm their written 1 2 3 4 5 

^msuni cation skills* 

152) 17> Know what level of basic skills 1 2 3 4 5 

profieiency is required in tte 
jobs of interest to them* 

(S3) 18* Gain confidence in their 1 2 3 4 5 

ability to apply basic skills 
to eon^lete tasks and to solve 
problems aroimd them. 



(62) 



(64) 



ERIC 



. Of UttlB 

Helpful or No Help 



(54) 19. McQine acquainted with a 

broad range of resources to 
use in gathering infoCTiation 
for work and decision making, 

(55) 20* Communicate COTfortably With 

adul ts. 

(56) 21. T^e res]^n3ibility for their 

own actions* 

(57) 22, Become more open to ideas 

^d values different from their 
own . 

(58) 2 3p Use information obtained 

through dire at esqjeriences 
in making daeisions. 

(59) 24, Fee], prepared to accept adult 

responsibilities , 



(60) 25. I€iat factors, if any, ha^ you seen this year that are contributing in a 

major way to the success of the EB^ program? 



26. What obstacles, if any, have you seen this year that are limiting the 
success of the EBCE program? 



27. In what areas do you feel EBO students hav^ made the greatest growth 
this year? vmy? 



13. 



year?"wh^r^ ^^"^ "^^^ students have made the laast growth this 



9, What effects, if any, do you feel the IBCE progrm haa had on the 
regular high school program? Why? 



What effects, if any, do you feel the EBCl program has had on the 
community? I^y? 



m^t ch^ges, if any, would you suggest in the EICE program for next 
year? 









Ifliat typmB 
uieful was 


of tachnical 
each typm of 


a^sistmee did you receive from mBML? How 
tectaical assistance? 







What additional types of technical assistanue, if any, did you requeat 
this year ttm NHBIL that you did not receive? 
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STUDENT APPLICATION/BACICGROUND QUESTIONNAIRE 



Purpost 

mmn Gtudenta formally apply to enter KBCE they v/ill be asked to 
Goniplete a brief application form containing information about 
.family baakgromd, students' previous employment history, short- 
and long-range edusational work goals, past involvement in 
high sahTCl and aomunity activities and reasons for wanting to 
ranter the IBOE prbgram. mis information will be used to describe 
the stucaents entering IBCT and also to compare changes over the 
year in areas such as educational and occupational aspirations, 
A modifioation of this questionnaire, deleting items referring to 
EBCEI, cw be adtiiinistered and used to describe the comparison group 
students. 



Directions for Data Collactlon 

This application/background questionnaire combines thm raquirement 
for parental permission for a student to participate in EBCT with 
a brief baakgroimd information section. This instrianent is 
completed individually by students as ttey apply for entry into 
the program. Students should be told that the information collected 
can be used to help the project staff know them better ^d will be 
held in confidence. TMs form takes atout 20 minutes to COT^lete. 



EBCE STUDENT APPLICATION 
Student Data 



Ag e Date of Birt h / / Sex: M f Gfide Level 

Month/Day/Yeaf (Circlt one) 

High School Counselor (1f applicable) - 



Address. ■ Tel ephorie_ 

street 



Glty state zip 

In Eniergency, Contac t [ Telephon e" 



Parent (s) or Guardian (s) 



Parent(s)'s or Guard1an(s)'s Oceupatlon(s) 



Describe in a paragraph your reasons for wishing to be In EBCI: 




PARENT PERMISSION 

Date. ^ 

I have discussed the EBCE program with 

and grant permission for his/her full parti cipatlon In EBCE. 



signmturs 
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(S) _ _ Studiiit Numbif ^ ^ Date 



Studtnt Backflfound Qusstionnalfi 

PlemMa Qhwak the mswmrm that dmBari^ you^ or writm in rmMponsm& whtin 
mmkmd to do so^ Ignorm the nm^jtm pmrmnthm&mM in thm Imft mrgins 
whiGh mrm for data proaemsing purpomms m%ly* 



1. Hiva j^ou ivtr worked for pay eutsidt oflymr hm^ on ^ regular basis? 

(If no^ ignon qummtim S mnd go directly to qu^Btian J^/^ 



(7) L □ Yas 



No 



2, If you answarad yts to question l* what types of work do you do now 
or have you done 1n the pait? 



(8) _ 

! 3, What do you expect to be doing one yB^r after completing high school? 



I (ohm^k orm or morm miMwmrB) 

(9)1 1. nw6rk1ng_ full -time 

(IQ) 2. Entering an apprintlcashlp or on-the-job training pfogram 

(11) 3i Q Going Into regular mnitary lervica or to a service academy 

(12) 4, Q Attending a vocational, technical, trade or business school 

(13) 5. Q Aittending a junior or community college 
(1*) 6, LJ Attending a four-year col lege or university 

(15) 7. Working part-time 

(16) 8, Q other (travel, take a break) 

(17) 9. rn 1 1 have no idta what rn be doing 



^PlmMMm ignorf thm nmrbmrs in paranth&sea in tho left mr^ln. Thmmm mmbors 
MXm for datm \ proaemming purposma only* 
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4. Hew far do ym plan to pursue your formal education? (shsek ©ne> 



1. 
2. 
3. 

4. 

5. 
6. 



n Don't plan to finish high school 

Graduatt from high school 



_ . . . I / . 

High school plus ona or two years of colleoe, comniunity 

coll tga or special training , ' ^ 

High school plus thraii or iiiore yearo of coll age. comwunlty 
college or special training 



Graduate from four-^ear college 



Sraduate or prof ^si ona 1 training beyond college 



/ 



List two jobs you feel^you would like to hold after completing your 
education, it as specific as possible, (for example, say "a mechanical 
drafter" rather thtfi "working at National Engintering^'). 



2. 



Have you observed or directly worked at either or both of the two 
preferred Jobs Itstid for quiitlon S? 

I havi obiirved or workad at bwth jobs 

Z. Q I hivg obiarvid or wofkid at ona of thast two Jobs 

I have not ©bsifved or worked at either job 



sure art you of steps to prapart for and enter the job 
would mast like to hold after graduation? 

1 I — I # 
1* ] I Do not know where to begin 

2. Have some Idea ^ 

3. Q steps pretty clear 

4* Q Strips quite clear 

Do yoM fiil you will be a, 
job? 

1. □ Yes 
2* Q Not sure 
3. [r] Probably not 



you 



to complete the necessary steps for this 



9. What are the occupations of youf father and mothar? 
' Father's occupation 



Mother's occupation 



10. What high school activities (such as choir or basketbtll ttam) did 
you pirtlclpati In the year before enterlno EiCI? Write "noni" if 
you did not participate in any. If you participated in sports, 
list the actuil names of the sports. * 



11. What specific high school activities, if any, do you plan to 
participate In this year? , 



12. List your hobbi is or recreational activities. 

_ K 



13. Approximately how many pamphlats.brochurei, manuals or magazine 
articles did you read this past school year? 



1. Q None 
1 to 6 



3. 
4. 



□ 6 to 10 S. Q 21 to 30 

Q U to 20 6. Q More than 30 



14. Are you 

1 



White 
Black 



3. Q Oriental 
4. 



5. 
6. 
7. 



Native ^erican Indian 
Nativi Alaskan 
Other (spicify) 



Spanish descant (ChlcanOi 
Puirto Rican and so fofth) 
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6. n Collegi graduata (four-^ 
ytar dagrii) 



What is thi highist lival of fortnal aducation your fathar has complatid? 

1. □ Nona 

2. [ I El amanta ry School 

3. Sonia high school 

4. Q High school graduata 



7*; 
8. 
9. 



CI Soma graduate work 
Q Advancad dtgrta 
Do not know 



S, j I Som poitsacondary (for ixampli* 
I soma collega, jr, conage. bus- 
1 nasi school, trada or tachnieil 
schoDl) 

What Is thi highest laval of fortnal aducation your mothar has completed? 
L I I Nona 

i 

Elamintary ichool 
Soma high school 
[~| H1g|i school graduata 



□ 



Col lege graduate (four- 
year degroe) 

Son^a graduate work 



8, Q Adviincad degree 



Do not know 



Soma postsacondary (forexamplet 
soma collagt, ■ % collaga, bus- 
^ intss se^*^nl^ trade or tichnlcal 
schoe' 

How Important was each of the following factors In deciding to Join EBCE? 

(ajxalm onm nm^mi^ fbr mmmh qumBtim. For mxmplm^ "r'^not importmnt. 



L I wantidjiiora fraadom/independence 

2. I wanted to choose own learning 
style 

3. I wanted to learji sbout caraeys 

4. I d1u not like my pfivious school 

5. I wanttd to prepare for a job 

6. I was borid with school 

7. I heBrd thi EICE program was aasy 

8. Othtr (spgcify) , ■ 



sot 

Jmportant 

1 
1 



SxtrmmBlg 
Important 



2 
2 



1 
1 
1 
1 
1 
1 



2 
2 
2 
2 
2 
2 



3 
3 

3 
3 
3 
3 
3 

3" 



4 
4 

4 
4 
4 
4 
4 
4 



5 
5 

S 
5 
5 
S 
5 
5 



STUDENT END-OF-YEAR QUESTIONNAIRE 



Purpose 

This questioimaire is dasigned to (1) follow up on questions asked 
on toe Student Bad^growid Questionnaire atoinistered at the 
beginning of thm year to aasese any ah«ige that mght have 
Qceurred dicing the year, (2) assess student Imowledge ^out job 
trends and related Information and (3) collact data on student 
reflections about their school/EBCT e^^erienaes. It can be used 
with eonparison group students by anitting items that relate only 
to EBCE* 



Directions for Data Collection 

This questionnaire can be atainistered to EICE students as a 
group during the postteet session, approximately one month prior 
to the end of the sdiool year. 'This allows time for students 
absent from the group testing session to complete tMs questionnaire 
on tiieir omi and twcn it in to a staff member. This is not a 
timed test* Students generally oOTplete it in 20 to 40 minutes. 
Since the intent of the ingtrOTent is to obtain students* opinions 
rather than to assess their reading skill, it is useful for the 
projS' staff to identify any students who are partioularly poor 
reade. j mnd have somone separately read tiie questionnaire aloud 
to these students as they oomplete it* 



11 
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6 2 Name . D ate 
School District 



^Student ID Number 



EBCE STUDENT END-OF-YEAR QUESTIONNAIRE 

You are rmmdy to aomplmte a ymmr of parti aipmtion in your Bxperimnm- 
BMMed Cmreer Eduamtim (EBCE) Progr&m. This questionnaire msks s&m 
of tte questions that you may ha^m been askmd in Septmmber and 

addm som mw ones that cover your aaremr plans , personal mxperimnaes 
and knowledge ^ut thm world of wMk, Xf you haw any questims while 
you arm aomplmting the survey pleasm ask for assistance* Ignore tha 
nun^rs in parentheses in the left margins* Thesm nui^rs are for data 
proaessing purposes only* 

1* ■ What dd y ou as^act to be doing ona year after completing high school? 
l.lfj Working full-time 

2 Entering an apprenticeship or on-the-- job training program 
a^y^ing into regular ndlita^ service or to a service academy 
4^ LJ Attending a vocational, tectaical, trade or business school 



Attending a jmior or commmiity collega 
fi^QAttan^ng a four-year oollaga or university 

Iferking part-tiina 
8* n Other (travel ^ t^e a break) 
9.ni have no idaa wl^t, I'll be doing 

How far do you plan to pursue your fomal aducation? (chmak one) 

l,n Don't plan to finish high srhool 

S^Ooraduate from high school 

school plus one or (two years pf college # consnmiity 
college or special training 

4.^^ High school plus torae or more years of college? conBnisiity 
collage or special training 

5 * ^Graduate front four year college 

6«nGraduate or profesoicmai training bayond collag| 



*Nttmtmra in pmrmn^esis ar« for key punahing pur^mms only- 



List two jobg feel you might like to hold after COTipleting 
you^sedueation. Give the voaation rather than location* (For 
mxmmpl^, mmy "m ^mftmr" rather thm "woMng at Nmtlmal 



1. 
2. 



Have you obserwd or direGtly worked at either or both of the two 
preferred jobs listed for question 3? 

I observed or worked at both jobs 

2.ni c^served or worked at one of these two jobs 

S^Dl did not observe or work at either job 

How sura are you of steps to prepare fur and enter toe job which 
you would most like to hold after graduatick? 

^o not know wheie to begin 
^^^^Have some idea 

steps pretty clear ' . 
4«Q Steps quite clear 

I 

Do you feel you will be ^le to conplete tte necessa^ steps for 
this job? 

l.nYes 

2* ^D^ot sure 

3.nProb^ly not 



l^at aspects of your learning es^erience ^lis year (if my) 
influenced yow choice of potential careers? (Ctmak as mmny as 
Apply.) 

l.^None 

2»Ql talked to teachers or a. ootmselor ^out choices 
3*ni talked to people who work at the jobs 
^•dl talked with relative or friends about my choices 
S.Ql had eKperience in observing or trying out the jobs 
S.Ql read abo^t the jobs 
other (plmAse wri te in) 



Are there any jobs that last J^ar seemed interesting that you 
ftw feel do not matoh your interests or abilities? 



es 



If yes# list these jobs 



Iti^ do you now feel .tfat the job(s) no \onger match your 
interests or i^ilities? 



^^at aaused you to ohange yoiir irdn^ ^out the job(s)7 (Chmak 
om or morm of thm ^oJIoi/ing.^/ \ 

1.^] Advice from teachers or a counselor 

Advice from relatives or friends 

Ss^^Advic'e from soTOone who works at the job(s) 

Information I haw read about the job(s) 

5- ^Dila^arience in observing or ti^ying out the jc?b{s) .<< 

interests have chmged 

don ' t know 

Other (plmAsm stmtm your rmABon) 



il6 



(30) 
(31) 



On the grid bmlow plmmsm aircle a nimbmr from 1 to 5 to indicMte how 
hBlpful you feml EBCE has bmen to you in Tmaahing maah objective* 
(For mxmmpl&, if you fmml EmE wmm vmry hBrpful - 1 ^cie 5, if 
mdmrmtmly helpful airale and if little or /jl i^uip cirulG l.j 

How helpful to you fa^l your ebCT es^rienMS tois year have been in 
assisting you to— 7 



9. 
10, 



Of Little 
or No Help 



solve pTCblems logically. 



undarstand the role of 
scienca in our society today. 



1 
1 



3 



/ 

/ 

/ very 

/ 

4 5 

5 



(32) 11, undarstand mora about 

yoiirself • 



(33) 12* get 3lmg with others, 

(34) 13* understand tha democratio 

process* 



/ 

/ 1 



4 
4 



(35) 14. ^velop your own creativity. 1 2 3 4 S 

(36) IS. leaTO how your interests 1 2 3 4 5 

and Militias fit into \ 

potential careers. \ . 

(37) 10* laarn how society's valuis^ 1 \ 2345 

\ the govermnent and the 

aconon^ affect the world of 

wort* * ' . , 



J44) 



learn %hat to look at \^^n 
cons i da ring a job. ^ 

learn Jic^w to find and /^e 
a job 




*^ Of Little . very 

* . or No Hmlp - Hmlpful 

(45) 24. know what level of b^sic 1 2 3 4 S 

/skills proficiency is 
/ raquirad in the jobs of < 



1 



interest to you* 



(46) 25 • gain con fi da nee in your 
1 / ability to apply basic 

I skills to complete tasks 

/ and to solve problems 

/ aground you. 

(47) / 26. become acquainted with a 

broad range of resources 
to use in gathering 
information for work and 
decision making* 

(48) 27, communicate comfortably 

with adults, 

(49) 28* take responsibility for 

your own actions* 

(50) 29. becora^ more open to ideas 

^d V'aluas different from 
your own. 

(51) 30. use information obtained 

through diract experiences 
in making decisions* 

(52) 31, ^ feel prepared to accept 

/ adult responsibilities* 

(53) 32. /ould you rate tfts overall quality of your EBCE program? 

Pooxf \ ExamllmiA^ 

1 2 3 4 -5 

(54) 33. If you had it to do over again § do you think you would decide 

to participate in BICS? 

/ ■ ^ ' 

// D&fin£tmly no Dmfinit&ly ymm 

/■/ 1 2 3 4 5 

/ /■ ' ^ • 

(55) , / 34* I|i EBCB have you felt tiiat ^u could progress at your own rate? 

Definltmly no Dmfinit&Xy yes 

1,2 3 4 5 



A is 



in eonparison with tha regular high sehool program, how much 
opportunity did EBm provide you foil leaning ^ut occupations? 



Much Imms 

I 1 



^bout smnm 

2 3U— -^^^4 



5 



In ccffi^arlson with €& wgular high school progrra, how much 
opportimij^^Ma EICT provide you for gansral learning? 



MQqh l&BB 



Mout saine 

3 



Muah more 
5 



In oompttrison with past es^arienoas in tha regular high school 
program, how motivatad are you to leam in BBCE? 

Mueh less About same Muah more 

\ I 2 3 4 ' 5 

I ■ 

What courses, if awy* have you taken this year at the high school^ 
a coBinunity oollage# en?ployar slta or elsawhare? (Pleasim list 
any courses and whmxm thmy wmrm takmn^) 



Name of bourses 



Hhere Did You Tahm This Cour aa? 



This year, approKiinataly how many pamphlets, brochures, manuals 
or maga'sine articles did you read? \ 

I ^ \ 

. 1. I None ^ 4. 11 to 30 \ 

2- I 1 to 5 21 to 30 \ 



3 •/ 6 to 10 6. More than 30 

1^ the 



Dura.n^ the school year, approximately how m«iy visits did you 
ro^e to the following commmiity resources? ' f^rite in th& 
nmibBt' next to mmah aoamimity rmsourae. Put "0" if you did not 
vi&iti a pmrtiaalmr resource J 

I Nunber of Visits 

#i^lic Mbraries ^ ^ 

iit^eimig 
iCourtg 



Public Hl. nga 
Colleges or IMiversitlaa 
State Legiiilature 



Check whether you a^ree or disagree with each of these statements 



(67) 

(68) 
(69) 

(70) 

(71) 

(72) 

(73) 



(74) 
(75) 

(76) 



41. 

42, 
43, 

44, 

45, 
46, 

47, . 



Most persons remain in the 
same job throughout their 
adult lives. 



48, 
49. 



50. 



Few wOTCH work ^utside of 
the hoTO after marriage* 

Lass thm one-third of all 
job openings -require a 
collage ^gree. 

Most people have 'thi ability 
to do well in any job if 
they set their mnds to it. 

There is only ono "right job^V 
for a person in terms of his/ 
her ^ilities* 

Thm unemployment rate of 20- 
year-olds in the labor 
markat is usually less thm 
the rate for other adults* 

The State anployment Se;fvice 
Office provides free 
information about job 
openings and job training 
programs * 

Apprentices are paid while 
they learn. 

Thm English and matt skills 
of frastuien are about the 
same from one college to 
another* 

Ten years from now most jobs 
will requdre fowc or more 
years of college* 



Agree (1) 

□ 

□ 
□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 



DisMgree (2) 

□ 
□ 



□ 
□ 
□ 

□ 

□ 
□ 

□ 



(77, 78) ; What do you thirLH are Uie two greatest waakngsaea of the EBCE 

program? (Check the responsaa you faei^re most appU cable or 
wri te in your own reBponsm . ) 

oma students can't handle the freedom 

roblema in the organization/staffing of toe program 

3. 0 Students not receiving sufficient training in basic ski lis 
or survival skills 

4, n Inadequate supervision of students on job sites 

5, nLack of a variety of job sites to meet students' interests 

6, ^] Other (please write in) 



(19, 80) 52* What do you think are the two greatest strengths of the EBCE 

program? (Check ttm responses you feel arm imst applicable or 
write in your own rasponse .) 

Good alteriiative to a risgular school prnqram 

2. n Quality of the staff 

students learn ateu't a variety of careers 

4. n Students learn about "real life" situations aiid 

re sp on sx bi 1 i^ty 

5. n Good way of getting students to learn 
5 . ^3 Ejcp^riense in wrking vdth adults 

Individual treatment of students 
8* n Other (please write in) 



^.-~^^s^^ — 
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STUDENT PROJECTS EVALUATION FORMS 



Purpose 

i 

These two forms are used to obtain ^ indapandent evaluation of 
students' Life Skills pjrojects and resulting reports for a random 
san^le of students in EBCE* ^ l^is serves as a primary method for 
assessing the Life Skills dimension of EBCE. The evaluation can 
be done by the program evaluator or a curriculmi specialist 
external to the IBCE program. Form A evaluates the extent to which 
an individu-.^ student's project meets the objectives of the life 
Skills area, incorporates the use of Basic Skills, relates to 
Career Development outcomeSf uses community si-:es productively^ 
utilizes a variety of resources in an integrated manner, capitalize 
on a student's interests, contains adequately specified performance 
criteria and is comprehensive * Resulting reports are assessed 
in terms of ability to conmimiicate, technical writing quality, 
achievement of specified criteria levels, integration of a 
student's es^eriences and an overall rating. Form B ^ /aluates the 
total set of p^jects completed by an^ individual with respect to 
use of a variety of resources, creation of a variety of outcomes, 
the challenge of the projects to the student and value of the 
projects in oonigparison wito the amount of credits obtained. 

Directions for Data Conactlon 

A random sanple of five or more EBOI students should be dra; , with 
an alternate list of additionally sampled students to serve as 
replacements for individuals in toe prima^ sarnie in case any of 
these students leave the program. The learning mmagers should be 
oriented to the pu^ose of the independent evaluation and given 
the names of the san^led students so that tiieir projects and 
reports cm be saved after the learning manager has evaluated 
each and discussed it with the student. Student permission should 
also be obtained for having an outside person read his 'or her 
projects and reports* In order to judge the appropriateness of a 
particular project to an individual student it is important to 
provide a brief profile of each sampled student to the person 
rating these ^tadent projects, A one or two^paragraph description 
of each student by tiie project staff is very helpful and should 
include toe student's grade level, sex, career and personal 
interests, educational and occupational aspirations # reasons for 
joining EBCl ond pretest scores in Basic Skills ind on attitude 
measles # 



(7, 8) 

(9^11) 
{12} 



EBCE INDIVIDUAL STUDENT PROJECT EVALUATION (FORM A) 



student's Name 



Li fa Skills Area_ 

Project Title 

Starting Date 



Hater 'a Name 



Site 



Predesigned^ 



or Individual 



Completion Date 



Total 



Time (days) 



(13) 



Project Evaluation 

1. Arm tile project activities designed to help the student meet aa 
speGifia objectives of this Life Skills area? (CiralB one) 



Deflnl tmly 
No 



1 



Coimnentsi 



UndBcided 
5 



DBfinitmly 



(14) 



Do the project activities maaningfully inco^orato the use of Basic 
Skills in a way to promote growth in oommmii cations * reading and/or 
mathematics? 



Mfinl tely 
No 



Und&ci ded 
4 5 6 



Dsflni tei^ 
Yes 

9 



(joitinents ^ 



^pImsb ignorm numbars in pArmnth&sms in thm Imft mrgin, ThmBe 
nim^mrB are for data proamsEing puvpomes only. \ 

■\ 

1S3 



3, im there a relaW.onship between project activities and Cait ex 
Development outCOTS objectives? 

Definitely Undeaidad Dmfinit&ly 

^« yes 

1234567 8 9 

CoBiaents i 



4* Was the en^loyer or conmiunity site used productively? (Check iff 
employer or aonmimity site is not usr ii ; 

DBfinltely Undeaidma Definitely 

Yes 



No 
1 

CommntBi 



5, Do the project activities utilize a variety of rasoaraea in an 

integrated and meaningful way? 

Definitely^ Uhdecided ^finitely 

res 



No 

1 2 3 4 5 6 



CoQiments t 



154 



6. Is the project designed to capitalise on student interahts? 

Definitely mdmcldad DefinitBly 

^° Yes 

(18) 12 3 4 5 6 . 7 8 9 



?• Are the perfciOTan» Griteria specif ieei adequeitely? 

Definitely mdacidBd DefinitBly 

res 

(1^) 12 3 4 5 6 7 8 9 

Cororontsi 



8, Are the activities and the criterion levels for those activities 

apprdpriate for tiie student's abilities? (C^ak if ariterion l&vmls 
are not spmaifi&di Q ; 

Defini tmly Undeai ded Dmfini tal y 



No 

(20) 1 2 3 4 5 6 

Ocmentsi 



Ybs 



9, le the project sufficiently cwnprehensive and con^lete? 

Dm finitely Undeai dmd Dmfini tal y 

No Y&B 

(ai) 1 > 3 4 5 6 7 8 9 

Ccfiraents I 



ERIC 



10. Ii^at is the quality of thm laarning nmnager's written evaluation? 

(Check if thmre were no written aommntsi ) 

Low Vndeaided High 

Quality ' Quality 

123456 7 89 

Ccamiientsi 

Product Evaluation 

11. Doas the student product have the technical quality (e.g, legibility^ 
gr^iQar# clarity, etc) ttat you would esqpact of a student of his/her 
ability? 

Daflnltmly tnidmald&d DBfinxtely 

No yms 

X 2 3 4 5 6 7 8 9 

ccmiBnts ; 



12, Does the student produet oowmunicate wll? 1 

Definitely Mdeaided Dmfinitmly 

iyb ' . Yes 

1 2 3 4 5 6 7 I 8 9 

Can^nts i 



iS6 



13' Does student product indicate that he/she reacted the criterion 
level speGifiad for each activity? (Chmak if aritmrion levml not 

DBfinl tmly Undmaided Definitely 

Fes 

123456789 

Coni^nts I 



14. How would you rate Uiis product in tems of the student's demonstratad 

ability to integrate his/her experience? 

No mdiaation undBaidmd DeflnitBly 

of intBgrmtlon IntBgrated 

1 2 3 4 5 6 7 8 9 

Coments % 



15, Given this student's ability, what is your overall evaluation of 
the product? 

D^flnitmly Undeaided DBfinitmly 

NegAtivm PoBitivm 

1 2 3 4 5 6 7 8 S 

Coianvints i , 



is? 



EBCE CCWBINED STUDENT PROJECTS EVALUATION (FORM B) 



Student's name 



Site 



Rater's name 



Projects Reviewed (Writa in th^ bOK&s the numbBr of predesigned and 

individual proj&ats reviewed^) 

Pradasignad Projects Life Skill Araa Individual Projects 



(28) • 




□ 


CSeative Davelopinent 


(29) 


□ 


(30) 




□ 


Critical Thinking 


(31) 


□ 


(32) 




□ 


Punotional CitizenBhip 


(33) 


□ 


(34) 




□ 


Personal/Social ^velopment 


(35) 


□ 


(iS) 




□ 


Soience 


(37) 


□ 


(38) 




□ 


Basic Skills - MathematiQs 


(39) 


□ 


(40) 




□ 


Basic Skills - Communications 


(41) 


□ 


(42) 




□ 


dther (list) 


(43) 


□ 


(44) 




□ 


Othar 


(45) 


□ 


(46) 




□ 


Other 


(47) 


□ 




Overall Pi 


roject Evaluation 







1* Do the projaets enGouraga thm use of a variety of rasources 

such as printed and A/V materials, public agencies, people, 

natural enviroment, local codes and laws, local avents, ate? 

, Definitml^ mdmaidmd Dmfinitmly 

to ) Ymm 

(48) 1 2 3 4 5 6 7 8 9 



*Pleaa# Ignorm the nishbers in pmrmnthmsmB In the left margin, Thmsm 
nm^mra mrm tor data proc&msing purpoMms only* 



158 



2, Do the projects encourage producing a variety of products, such as 
witten reports* oral pri^santations, tape recordings, handicrafts, 
charts, citizen partia/tpatiQn, activity performance, etc? 

Dmfinitely mdmclded Dmfinit&ly 

No YBS 

(49) 1 2 3 4 5 6 7 8 9 

OomBientai 



3, As a whole, how challenging were these projects for this student? 

Not Undmaldmd Very 

ChAllBngIng Challenging 
at All 

(50) 1 2 3 4 5 6 7 8 9 

Cpsaoants i 



4» tea the projects sufficiently challenging to this student to be worth 
the credit obt^ned? 

DBfinitaly Undeaidmd - Smflnitely 

No Y^s 

(SI) 1 2 3456789 

Comments i 



:* 

o 
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5NDEX 
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CURRICm*UM/FROGHAM FEATURIS 



Basic skills . , . . 

Career ^velopment / , . , 
Coo^etancias . * . , 

Life skills 

Program year action lone^ 
Student projectfr * * . . , 



5, 39, 40, 60, 61, 6t 
...... 5, 62, 70 

^ . * . viii, S* 61 
........ 6, 61 

7 

, , , a 



EBCE 



Demonstration project 
Part D 

Pr<^r^ description 



V, 6 

7 

V, 6 



/ 

Data analysis 
Data /col lection 

Elas4nts of a pl«i ...... ^ 

Instrument S#lection/Developnmnt 
F^porting * j ......... . 

;fimeline ^ ..... . . . 

/ 

fpCOSING THE EV^UATION 



6, 56 

3| 33, 34, 47, 52, 51, 71 

, . 6, 21, 30, 33, 43, 46 

6, 8, 32, 3S-42, 69, S3 

57, 58 

35, 49 



/ 



Decisiort alternatives . . , 
Evaluation questions . . * 
In forma ti^ priorities . . 
Frogri.ii goals and objectives 
Use of infomation . , . 



....... 2S 

, . . . . 26, 27 

. * . . . 28, 29 
.... 8, 22, 23 

23, 36, 60, 64, 65 



CurricUliSR s Tnstruction * . 
Eniployer/Coimnmity Resources 
Mwagement & Organisation 
Stu«ient Services ■ , . , . . 



i^ 
vii 
ix 



INSTRUMENTS 



Directions for administering 
NWHSL^ devil aped instrtmnta 
Ordering inetrurnents . * * 
Standardized instruments . . 



71-80 

I ■ 83 

69, 70 
38-41 



POLICIES AND GUIMJLTNES 



Budget and resources • « \ 
pvaiuation responsibilities 
Legislative gi^ da lines ^ . 
Protection of h^ian st^jects 
Purposes for evaluation . . 



11, ^L, 35 

11, 19 

11, 13, 16 

8, 11, 17 

11, 13, 14 



TYPES OF EVALUATION 



fonn^tive evaluation 
SuDEiative evaluation 
Thi^-d party evaluation 



14, 26, 65 
^ ^ p » 8, 65 
^ * * * 8, 58 
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